[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:33:12 dist-test-slave-ctbw krb5kdc[3029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:33:12 dist-test-slave-ctbw krb5kdc[3029](info): set up 2 sockets
Jun 20 11:33:12 dist-test-slave-ctbw krb5kdc[3029](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 11:33:13.870971 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:33:13 dist-test-slave-ctbw krb5kdc[3029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:33:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:33:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:33:13.989892 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:37463
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:37463
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:14.078645 3042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:14.078924 3042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:14.078966 3042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:14.080196 3042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:33:14.080266 3042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:14.080302 3042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:14.080348 3042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:33:14.080379 3042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:33:14.081851 3042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:37463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:37463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:14.082120 3042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:14.082376 3042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 11:33:14.087647 3042 server_base.cc:1034] running on GCE node
W20260620 11:33:14.087616 3047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:14.087638 3050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:14.087657 3048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:14.088147 3042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:14.088440 3042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:14.089599 3042 hybrid_clock.cc:648] HybridClock initialized: now 1781955194089586 us; error 40 us; skew 500 ppm
Jun 20 11:33:14 dist-test-slave-ctbw krb5kdc[3029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955194, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:14.092175 3042 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:33:14.092639 3042 webserver.cc:460] Webserver started at http://127.2.243.190:46253/ using document root <none> and password file <none>
I20260620 11:33:14.092875 3042 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:14.092947 3042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:14.093104 3042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:14.094152 3042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "b49edd58742847e78c1a74d543ccd2f6"
format_stamp: "Formatted at 2026-06-20 11:33:14 on dist-test-slave-ctbw"
server_key: "343fbd031a9081bc0bfbfebc8bce24e3"
server_key_iv: "610f88664960e85044a39f034b6f4837"
server_key_version: "encryptionkey@0"
I20260620 11:33:14.094509 3042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "b49edd58742847e78c1a74d543ccd2f6"
format_stamp: "Formatted at 2026-06-20 11:33:14 on dist-test-slave-ctbw"
server_key: "343fbd031a9081bc0bfbfebc8bce24e3"
server_key_iv: "610f88664960e85044a39f034b6f4837"
server_key_version: "encryptionkey@0"
I20260620 11:33:14.095952 3042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:33:14.096768 3057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:14.096989 3042 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:14.097105 3042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "b49edd58742847e78c1a74d543ccd2f6"
format_stamp: "Formatted at 2026-06-20 11:33:14 on dist-test-slave-ctbw"
server_key: "343fbd031a9081bc0bfbfebc8bce24e3"
server_key_iv: "610f88664960e85044a39f034b6f4837"
server_key_version: "encryptionkey@0"
I20260620 11:33:14.097179 3042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:14.109393 3042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:14.111829 3042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:14.112028 3042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:14.116468 3042 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:37463
I20260620 11:33:14.116503 3109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:37463 every 8 connection(s)
I20260620 11:33:14.116866 3042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:33:14.117692 3110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:14.120062 3110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6: Bootstrap starting.
I20260620 11:33:14.120628 3110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:14.120918 3110 log.cc:824] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:14.121610 3110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6: No bootstrap required, opened a new log
I20260620 11:33:14.123831 3110 raft_consensus.cc:348] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49edd58742847e78c1a74d543ccd2f6" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 37463 } }
I20260620 11:33:14.123976 3110 raft_consensus.cc:374] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:14.123999 3110 raft_consensus.cc:729] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b49edd58742847e78c1a74d543ccd2f6, State: Initialized, Role: FOLLOWER
I20260620 11:33:14.124166 3110 consensus_queue.cc:260] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49edd58742847e78c1a74d543ccd2f6" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 37463 } }
I20260620 11:33:14.124230 3110 raft_consensus.cc:388] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:33:14.124255 3110 raft_consensus.cc:482] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:33:14.124311 3110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:14.124855 3110 raft_consensus.cc:504] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49edd58742847e78c1a74d543ccd2f6" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 37463 } }
I20260620 11:33:14.124975 3110 leader_election.cc:302] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b49edd58742847e78c1a74d543ccd2f6; no voters:
I20260620 11:33:14.124962 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 3042
I20260620 11:33:14.125181 3110 leader_election.cc:288] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:33:14.125205 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:33:14.125288 3113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:14.125415 3113 raft_consensus.cc:686] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6 [term 1 LEADER]: Becoming Leader. State: Replica: b49edd58742847e78c1a74d543ccd2f6, State: Running, Role: LEADER
I20260620 11:33:14.125478 3110 sys_catalog.cc:564] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:14.125494 3113 consensus_queue.cc:237] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49edd58742847e78c1a74d543ccd2f6" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 37463 } }
I20260620 11:33:14.125803 3113 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b49edd58742847e78c1a74d543ccd2f6. Latest consensus state: current_term: 1 leader_uuid: "b49edd58742847e78c1a74d543ccd2f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49edd58742847e78c1a74d543ccd2f6" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 37463 } } }
I20260620 11:33:14.125875 3113 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6 [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:14.126024 3022 external_mini_cluster.cc:1428] Setting key 1e15972930baab9621d1d496a1e40ec9
I20260620 11:33:14.126053 3115 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b49edd58742847e78c1a74d543ccd2f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49edd58742847e78c1a74d543ccd2f6" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 37463 } } }
I20260620 11:33:14.126122 3115 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6 [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:14.126704 3118 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:33:14.127425 3118 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:33:14.131960 3118 catalog_manager.cc:1347] Generated new cluster ID: 0a78d76d164a4e058bbf6385ce684a6d
I20260620 11:33:14.132015 3118 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:33:14 dist-test-slave-ctbw krb5kdc[3029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:14.140793 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:14.126929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40744 (local address 127.2.243.190:37463)
0620 11:33:14.127117 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:14.127127 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:14.129075 (+ 1948us) server_negotiation.cc:407] Connection header received
0620 11:33:14.129159 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:14.129163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:14.129203 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:14.129334 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:14.130813 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:14.132035 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:14.133118 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:14.133560 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:14.133685 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:14.137064 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:14.137068 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:14.137068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:14.137082 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:14.138826 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:14.139346 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:14.139349 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:14.139349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:14.139446 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:14.139706 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:14.139707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:14.139708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:14.139957 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:14.140098 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:14.140395 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 11:33:14.140526 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":60,"threads_started":1}
I20260620 11:33:14.159149 3118 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:33:14.159910 3118 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:33:14.168403 3118 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b49edd58742847e78c1a74d543ccd2f6: Generated new TSK 0
I20260620 11:33:14.168630 3118 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
I20260620 11:33:14.195904 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:37463
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:14.277236 3136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:14.277438 3136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:14.277462 3136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:14.278579 3136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:14.278622 3136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:14.278649 3136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:33:14.280189 3136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:37463
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:14.280457 3136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:14.280691 3136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:14.285703 3142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:14.285718 3144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:14.285735 3136 server_base.cc:1034] running on GCE node
W20260620 11:33:14.285707 3141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:14.286140 3136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:14.286336 3136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:14.287477 3136 hybrid_clock.cc:648] HybridClock initialized: now 1781955194287468 us; error 32 us; skew 500 ppm
Jun 20 11:33:14 dist-test-slave-ctbw krb5kdc[3029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955194, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:14.294669 3136 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:33:14.295249 3136 webserver.cc:460] Webserver started at http://127.2.243.129:37771/ using document root <none> and password file <none>
I20260620 11:33:14.295495 3136 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:14.295578 3136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:14.295725 3136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:14.296753 3136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "5df5910985a0478baf52425644dfc8f3"
format_stamp: "Formatted at 2026-06-20 11:33:14 on dist-test-slave-ctbw"
server_key: "28209e73cf13cfd33180ef25d740f822"
server_key_iv: "d05854df9f469df95a7fac473c51532e"
server_key_version: "encryptionkey@0"
I20260620 11:33:14.297137 3136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "5df5910985a0478baf52425644dfc8f3"
format_stamp: "Formatted at 2026-06-20 11:33:14 on dist-test-slave-ctbw"
server_key: "28209e73cf13cfd33180ef25d740f822"
server_key_iv: "d05854df9f469df95a7fac473c51532e"
server_key_version: "encryptionkey@0"
I20260620 11:33:14.298858 3136 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 11:33:14.299685 3151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:14.299897 3136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:14.299985 3136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "5df5910985a0478baf52425644dfc8f3"
format_stamp: "Formatted at 2026-06-20 11:33:14 on dist-test-slave-ctbw"
server_key: "28209e73cf13cfd33180ef25d740f822"
server_key_iv: "d05854df9f469df95a7fac473c51532e"
server_key_version: "encryptionkey@0"
I20260620 11:33:14.300035 3136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:14.318307 3136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:14.320436 3136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:14.320546 3136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:14.320721 3136 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:14.321048 3136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:14.321111 3136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:14.321152 3136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:14.321182 3136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:14.327024 3136 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:41433
I20260620 11:33:14.327525 3264 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:41433 every 8 connection(s)
I20260620 11:33:14.327796 3136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:33:14.331108 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 3136
I20260620 11:33:14.331187 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:33:14.331346 3022 external_mini_cluster.cc:1428] Setting key 020ab459e539e5f91baac50ffd6ad208
Jun 20 11:33:14 dist-test-slave-ctbw krb5kdc[3029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955194, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:14.338255 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:14.328513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:58363 (local address 127.2.243.190:37463)
0620 11:33:14.328576 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:14.328578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:14.328770 (+ 192us) server_negotiation.cc:407] Connection header received
0620 11:33:14.329291 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:14.329292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:14.329301 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:14.329365 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:14.330315 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:14.330838 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:14.332218 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:14.332566 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:14.332646 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:14.335227 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:14.335228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:14.335229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:14.335236 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:14.336885 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:14.337312 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:14.337314 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:14.337314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:14.337373 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:14.337621 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:14.337622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:14.337623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:14.337735 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:14.337812 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:14.338078 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 11:33:14.338167 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 11:33:14.338438 3267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:14.328480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:37463 (local address 127.2.243.129:58363)
0620 11:33:14.328642 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:14.328684 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:14.329190 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:14.329420 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:14.329424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:14.330004 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:14.330198 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:14.330203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:14.330995 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:14.330997 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:14.332141 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:14.332144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:14.332665 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:14.332666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:14.332782 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:14.333365 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:14.333376 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:14.335109 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:14.336990 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:14.336996 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:14.336998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:14.337225 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:14.337441 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:14.337441 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:14.337442 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:14.337553 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:14.337803 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:14.337809 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:14.337948 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 11:33:14.338056 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 11:33:14.338208 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260620 11:33:14.339195 3265 heartbeater.cc:347] Connected to a master server at 127.2.243.190:37463
I20260620 11:33:14.339273 3265 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.2.243.130@KRBTEST.COM; defaulting to no policy
I20260620 11:33:14.339494 3265 heartbeater.cc:511] Master 127.2.243.190:37463 requested a full tablet report, sending...
I20260620 11:33:14.340036 3074 ts_manager.cc:194] Registered new tserver with Master: 5df5910985a0478baf52425644dfc8f3 (127.2.243.129:41433)
I20260620 11:33:14.340739 3074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:58363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
I20260620 11:33:14.356202 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:37463
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:14.438014 3270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:14.438207 3270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:14.438230 3270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:14.439410 3270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:14.439450 3270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:14.439477 3270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:33:14.440960 3270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:37463
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:14.441191 3270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:14.441412 3270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 11:33:14.446216 3270 server_base.cc:1034] running on GCE node
W20260620 11:33:14.446187 3275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:14.446200 3276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:14.446225 3278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:14.446571 3270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:14.446772 3270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:14.448077 3270 hybrid_clock.cc:648] HybridClock initialized: now 1781955194448067 us; error 34 us; skew 500 ppm
Jun 20 11:33:14 dist-test-slave-ctbw krb5kdc[3029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955194, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:14.450577 3270 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:33:14.451048 3270 webserver.cc:460] Webserver started at http://127.2.243.130:34453/ using document root <none> and password file <none>
I20260620 11:33:14.451261 3270 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:14.451306 3270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:14.451442 3270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:14.452420 3270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "4991763d2e6c408a8bd0aec29d58273e"
format_stamp: "Formatted at 2026-06-20 11:33:14 on dist-test-slave-ctbw"
server_key: "874c4d34e0408c9c37f9e55f6d170164"
server_key_iv: "60e6f28cd5debd39de73b358a569ea00"
server_key_version: "encryptionkey@0"
I20260620 11:33:14.452754 3270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "4991763d2e6c408a8bd0aec29d58273e"
format_stamp: "Formatted at 2026-06-20 11:33:14 on dist-test-slave-ctbw"
server_key: "874c4d34e0408c9c37f9e55f6d170164"
server_key_iv: "60e6f28cd5debd39de73b358a569ea00"
server_key_version: "encryptionkey@0"
I20260620 11:33:14.454033 3270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:33:14.454800 3285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:14.454980 3270 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:14.455073 3270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "4991763d2e6c408a8bd0aec29d58273e"
format_stamp: "Formatted at 2026-06-20 11:33:14 on dist-test-slave-ctbw"
server_key: "874c4d34e0408c9c37f9e55f6d170164"
server_key_iv: "60e6f28cd5debd39de73b358a569ea00"
server_key_version: "encryptionkey@0"
I20260620 11:33:14.455144 3270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:14.466624 3270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:14.468918 3270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:14.469072 3270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:14.469311 3270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:14.469620 3270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:14.469652 3270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:14.469702 3270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:14.469735 3270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:14.475673 3270 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:46769
I20260620 11:33:14.475713 3398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:46769 every 8 connection(s)
I20260620 11:33:14.476018 3270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:33:14.480834 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 3270
I20260620 11:33:14.480962 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:33:14.481129 3022 external_mini_cluster.cc:1428] Setting key ad66671eca6aa6b61dd3cf75473d2b4e
Jun 20 11:33:14 dist-test-slave-ctbw krb5kdc[3029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955194, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:14.486380 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:14.476727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:58331 (local address 127.2.243.190:37463)
0620 11:33:14.476805 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:14.476808 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:14.476972 (+ 164us) server_negotiation.cc:407] Connection header received
0620 11:33:14.477438 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:14.477439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:14.477449 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:14.477510 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:14.478434 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:14.479004 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:14.479940 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:14.480124 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:14.480182 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:14.483021 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:14.483022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:14.483023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:14.483030 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:14.484912 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:14.485398 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:14.485399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:14.485399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:14.485449 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:14.485722 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:14.485723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:14.485723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:14.485817 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:14.485898 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:14.486255 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 11:33:14.486318 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 11:33:14.486670 3401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:14.476679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:37463 (local address 127.2.243.130:58331)
0620 11:33:14.476852 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:14.476886 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:14.477337 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:14.477582 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:14.477585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:14.478140 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:14.478347 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:14.478352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:14.479081 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:14.479082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:14.479860 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:14.479862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:14.480197 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:14.480197 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:14.480291 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:14.480899 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:14.480928 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:14.482532 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:14.484996 (+ 2464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:14.485001 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:14.485004 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:14.485323 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:14.485513 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:14.485514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:14.485515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:14.485661 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:14.485914 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:14.485920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:14.486098 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 11:33:14.486233 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 11:33:14.486419 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260620 11:33:14.487277 3399 heartbeater.cc:347] Connected to a master server at 127.2.243.190:37463
I20260620 11:33:14.487388 3399 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:14.487591 3399 heartbeater.cc:511] Master 127.2.243.190:37463 requested a full tablet report, sending...
I20260620 11:33:14.488070 3074 ts_manager.cc:194] Registered new tserver with Master: 4991763d2e6c408a8bd0aec29d58273e (127.2.243.130:46769)
I20260620 11:33:14.488591 3074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:58331
WARNING: no policy specified for kudu/127.2.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
I20260620 11:33:14.507046 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:37463
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:14.587049 3404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:14.587255 3404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:14.587278 3404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:14.588474 3404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:14.588521 3404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:14.588547 3404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:33:14.589977 3404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:37463
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:14.590231 3404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:14.590466 3404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:14.595468 3412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:14.595471 3409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:14.595706 3410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:14.596026 3404 server_base.cc:1034] running on GCE node
I20260620 11:33:14.596215 3404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:14.596426 3404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:14.597592 3404 hybrid_clock.cc:648] HybridClock initialized: now 1781955194597555 us; error 53 us; skew 500 ppm
Jun 20 11:33:14 dist-test-slave-ctbw krb5kdc[3029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955194, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:14.600056 3404 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:33:14.600522 3404 webserver.cc:460] Webserver started at http://127.2.243.131:38739/ using document root <none> and password file <none>
I20260620 11:33:14.600795 3404 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:14.600847 3404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:14.601018 3404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:14.602110 3404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "4facfb3d0f9a4aa5a10ed90642581a45"
format_stamp: "Formatted at 2026-06-20 11:33:14 on dist-test-slave-ctbw"
server_key: "968dbac2ede0bb167aa93e3ce403331b"
server_key_iv: "7be269c1f6eedcb97112e0be3437d8e5"
server_key_version: "encryptionkey@0"
I20260620 11:33:14.602412 3404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "4facfb3d0f9a4aa5a10ed90642581a45"
format_stamp: "Formatted at 2026-06-20 11:33:14 on dist-test-slave-ctbw"
server_key: "968dbac2ede0bb167aa93e3ce403331b"
server_key_iv: "7be269c1f6eedcb97112e0be3437d8e5"
server_key_version: "encryptionkey@0"
I20260620 11:33:14.603780 3404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:33:14.604449 3419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:14.604658 3404 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:33:14.604732 3404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "4facfb3d0f9a4aa5a10ed90642581a45"
format_stamp: "Formatted at 2026-06-20 11:33:14 on dist-test-slave-ctbw"
server_key: "968dbac2ede0bb167aa93e3ce403331b"
server_key_iv: "7be269c1f6eedcb97112e0be3437d8e5"
server_key_version: "encryptionkey@0"
I20260620 11:33:14.604817 3404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:14.631374 3404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:14.633437 3404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:14.633549 3404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:14.633746 3404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:14.634047 3404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:14.634083 3404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:14.634106 3404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:14.634119 3404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:14.640971 3404 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:42593
I20260620 11:33:14.641004 3532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:42593 every 8 connection(s)
I20260620 11:33:14.641433 3404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:33:14.641782 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 3404
I20260620 11:33:14.641867 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:33:14.642040 3022 external_mini_cluster.cc:1428] Setting key bca790e8c7ca913c50831416ce291931
Jun 20 11:33:14 dist-test-slave-ctbw krb5kdc[3029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955194, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:14.652141 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:14.642277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:44251 (local address 127.2.243.190:37463)
0620 11:33:14.642334 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:14.642336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:14.643080 (+ 744us) server_negotiation.cc:407] Connection header received
0620 11:33:14.643753 (+ 673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:14.643753 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:14.643762 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:14.643800 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:14.644761 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:14.645263 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:14.646329 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:14.646523 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:14.646620 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:14.649158 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:14.649160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:14.649160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:14.649169 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:14.650552 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:14.651095 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:14.651096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:14.651096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:14.651165 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:14.651437 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:14.651438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:14.651438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:14.651565 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:14.651643 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:14.651962 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 11:33:14.652040 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 11:33:14.652379 3535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:14.642263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:37463 (local address 127.2.243.131:44251)
0620 11:33:14.642465 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:14.642514 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:14.643636 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:14.643874 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:14.643878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:14.644459 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:14.644654 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:14.644659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:14.645390 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:14.645392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:14.646214 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:14.646216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:14.646640 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:14.646641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:14.646779 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:14.647421 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:14.647431 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:14.649040 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:14.650667 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:14.650674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:14.650676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:14.650996 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:14.651238 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:14.651239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:14.651239 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:14.651375 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:14.651634 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:14.651640 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:14.651781 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 11:33:14.651928 (+ 147us) client_negotiation.cc:240] Negotiation successful
0620 11:33:14.652110 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":89,"threads_started":1}
I20260620 11:33:14.652872 3533 heartbeater.cc:347] Connected to a master server at 127.2.243.190:37463
I20260620 11:33:14.652972 3533 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:14.653200 3533 heartbeater.cc:511] Master 127.2.243.190:37463 requested a full tablet report, sending...
I20260620 11:33:14.653653 3074 ts_manager.cc:194] Registered new tserver with Master: 4facfb3d0f9a4aa5a10ed90642581a45 (127.2.243.131:42593)
I20260620 11:33:14.654170 3074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:44251
I20260620 11:33:14.654911 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 11:33:14 dist-test-slave-ctbw krb5kdc[3029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955194, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 11:33:14 dist-test-slave-ctbw krb5kdc[3029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955194, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.243.129@KRBTEST.COM
I20260620 11:33:14.678038 3540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:14.670281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52840 (local address 127.2.243.129:41433)
0620 11:33:14.670476 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:14.670481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:14.670494 (+ 13us) server_negotiation.cc:407] Connection header received
0620 11:33:14.670521 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:14.670522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:14.670535 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:14.670601 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:14.671257 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:14.671994 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:14.672728 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:14.673086 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:14.673153 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:14.675526 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:14.675533 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:14.675533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:14.675539 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:14.676966 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:14.677281 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:14.677283 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:14.677283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:14.677336 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:14.677499 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:14.677499 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:14.677500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:14.677617 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:14.677731 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:14.677769 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 11:33:14.677867 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":113,"threads_started":1}
W20260620 11:33:14.678380 3179 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52840
Jun 20 11:33:14 dist-test-slave-ctbw krb5kdc[3029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 11:33:14 dist-test-slave-ctbw krb5kdc[3029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.129@KRBTEST.COM
I20260620 11:33:14.700311 3540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:14.693518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52856 (local address 127.2.243.129:41433)
0620 11:33:14.693597 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:14.693599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:14.693610 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:33:14.693717 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:14.693718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:14.693731 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:14.693791 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:14.694316 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:14.694804 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:14.695575 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:14.695769 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:14.695825 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:14.698003 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:14.698004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:14.698004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:14.698010 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:14.699305 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:14.699655 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:14.699656 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:14.699656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:14.699750 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:14.699920 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:14.699921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:14.699921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:14.700019 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:14.700105 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:14.700167 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 11:33:14.700222 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 11:33:14.700986 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 3136
I20260620 11:33:14.707238 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 3270
I20260620 11:33:14.711789 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 3404
I20260620 11:33:14.716049 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 3042
2026-06-20T11:33:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1897 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:33:14 dist-test-slave-ctbw krb5kdc[3547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:33:14 dist-test-slave-ctbw krb5kdc[3547](info): set up 2 sockets
Jun 20 11:33:14 dist-test-slave-ctbw krb5kdc[3547](info): commencing operation
krb5kdc: starting...
W20260620 11:33:15.754812 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:33:15 dist-test-slave-ctbw krb5kdc[3547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:33:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:33:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:33:15.873908 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:40683
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:40683
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:15.956337 3560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:15.956580 3560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:15.956616 3560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:15.957738 3560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:33:15.957804 3560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:15.957830 3560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:15.957861 3560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:33:15.957886 3560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:33:15.959406 3560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:40683
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:40683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:15.959671 3560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:15.959899 3560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:15.964644 3566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:15.964649 3565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:15.964686 3568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:15.964893 3560 server_base.cc:1034] running on GCE node
I20260620 11:33:15.965157 3560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:15.965391 3560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:15.966554 3560 hybrid_clock.cc:648] HybridClock initialized: now 1781955195966520 us; error 64 us; skew 500 ppm
Jun 20 11:33:15 dist-test-slave-ctbw krb5kdc[3547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955195, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:15.968982 3560 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:33:15.969398 3560 webserver.cc:460] Webserver started at http://127.2.243.190:39115/ using document root <none> and password file <none>
I20260620 11:33:15.969638 3560 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:15.969708 3560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:15.969856 3560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:15.970971 3560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "2d872f40f9c64e3e9fb0a35c99d9651b"
format_stamp: "Formatted at 2026-06-20 11:33:15 on dist-test-slave-ctbw"
server_key: "e29ee1e1ff193d462c7dd6819ee50b55"
server_key_iv: "7b91738fedf92f9ab976f802fb009b4a"
server_key_version: "encryptionkey@0"
I20260620 11:33:15.971331 3560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "2d872f40f9c64e3e9fb0a35c99d9651b"
format_stamp: "Formatted at 2026-06-20 11:33:15 on dist-test-slave-ctbw"
server_key: "e29ee1e1ff193d462c7dd6819ee50b55"
server_key_iv: "7b91738fedf92f9ab976f802fb009b4a"
server_key_version: "encryptionkey@0"
I20260620 11:33:15.972673 3560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 11:33:15.973397 3575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:15.973587 3560 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:33:15.973685 3560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "2d872f40f9c64e3e9fb0a35c99d9651b"
format_stamp: "Formatted at 2026-06-20 11:33:15 on dist-test-slave-ctbw"
server_key: "e29ee1e1ff193d462c7dd6819ee50b55"
server_key_iv: "7b91738fedf92f9ab976f802fb009b4a"
server_key_version: "encryptionkey@0"
I20260620 11:33:15.973764 3560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:15.985690 3560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:15.987740 3560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:15.987905 3560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:15.991669 3560 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:40683
I20260620 11:33:15.991711 3627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:40683 every 8 connection(s)
I20260620 11:33:15.992056 3560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:33:15.992869 3628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:15.995110 3628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b: Bootstrap starting.
I20260620 11:33:15.995622 3628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:15.995841 3628 log.cc:824] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:15.996484 3628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b: No bootstrap required, opened a new log
I20260620 11:33:15.997982 3628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d872f40f9c64e3e9fb0a35c99d9651b" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40683 } }
I20260620 11:33:15.998111 3628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:15.998139 3628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d872f40f9c64e3e9fb0a35c99d9651b, State: Initialized, Role: FOLLOWER
I20260620 11:33:15.998271 3628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d872f40f9c64e3e9fb0a35c99d9651b" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40683 } }
I20260620 11:33:15.998347 3628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:33:15.998368 3628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:33:15.998396 3628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:15.998646 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 3560
I20260620 11:33:15.998728 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:33:15.998907 3022 external_mini_cluster.cc:1428] Setting key c8b4cbcbd533176c0657fcabb4cf217f
I20260620 11:33:15.999127 3628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d872f40f9c64e3e9fb0a35c99d9651b" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40683 } }
I20260620 11:33:15.999212 3628 leader_election.cc:302] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d872f40f9c64e3e9fb0a35c99d9651b; no voters:
I20260620 11:33:15.999589 3628 leader_election.cc:288] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:33:15.999646 3633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:15.999776 3633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b [term 1 LEADER]: Becoming Leader. State: Replica: 2d872f40f9c64e3e9fb0a35c99d9651b, State: Running, Role: LEADER
I20260620 11:33:15.999850 3628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:15.999866 3633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d872f40f9c64e3e9fb0a35c99d9651b" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40683 } }
I20260620 11:33:16.000651 3635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d872f40f9c64e3e9fb0a35c99d9651b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d872f40f9c64e3e9fb0a35c99d9651b" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40683 } } }
I20260620 11:33:16.000661 3634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d872f40f9c64e3e9fb0a35c99d9651b. Latest consensus state: current_term: 1 leader_uuid: "2d872f40f9c64e3e9fb0a35c99d9651b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d872f40f9c64e3e9fb0a35c99d9651b" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40683 } } }
I20260620 11:33:16.000774 3635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:16.000813 3634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:16.001046 3643 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:33:16.001585 3643 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:33:16.003849 3643 catalog_manager.cc:1347] Generated new cluster ID: 31472a78783645d6924498928a6b295f
I20260620 11:33:16.003902 3643 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:33:16 dist-test-slave-ctbw krb5kdc[3547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:16.009281 3632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:15.999357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38190 (local address 127.2.243.190:40683)
0620 11:33:15.999534 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:15.999543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:15.999579 (+ 36us) server_negotiation.cc:407] Connection header received
0620 11:33:15.999699 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:15.999706 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:15.999760 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:16.000731 (+ 971us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:16.000832 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.002192 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.002957 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.003561 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.003681 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:16.005994 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:16.005998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:16.005998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:16.006010 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:16.007747 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:16.008137 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:16.008140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:16.008140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:16.008239 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:16.008474 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:16.008475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:16.008476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:16.008701 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:16.008875 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:16.008941 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 11:33:16.009075 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260620 11:33:16.037406 3643 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:33:16.038163 3643 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:33:16.045136 3643 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d872f40f9c64e3e9fb0a35c99d9651b: Generated new TSK 0
I20260620 11:33:16.045333 3643 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
I20260620 11:33:16.079699 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:40683
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:16.160102 3654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:16.160307 3654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:16.160331 3654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:16.161405 3654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:16.161439 3654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:16.161465 3654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:33:16.162829 3654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:40683
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:16.163021 3654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:16.163189 3654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:16.168164 3659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:16.168164 3662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:16.168241 3660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:16.168418 3654 server_base.cc:1034] running on GCE node
I20260620 11:33:16.168654 3654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:16.168869 3654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:16.170037 3654 hybrid_clock.cc:648] HybridClock initialized: now 1781955196170017 us; error 34 us; skew 500 ppm
Jun 20 11:33:16 dist-test-slave-ctbw krb5kdc[3547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955196, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:16.172731 3654 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:33:16.173255 3654 webserver.cc:460] Webserver started at http://127.2.243.129:33831/ using document root <none> and password file <none>
I20260620 11:33:16.173527 3654 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:16.173583 3654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:16.173764 3654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:16.174986 3654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "e657dff0f27041e3a3a11dcf0225b72b"
format_stamp: "Formatted at 2026-06-20 11:33:16 on dist-test-slave-ctbw"
server_key: "9f89799f53602cea70a3617417959a08"
server_key_iv: "18be159185426e086ae2d4368d93dda6"
server_key_version: "encryptionkey@0"
I20260620 11:33:16.175387 3654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "e657dff0f27041e3a3a11dcf0225b72b"
format_stamp: "Formatted at 2026-06-20 11:33:16 on dist-test-slave-ctbw"
server_key: "9f89799f53602cea70a3617417959a08"
server_key_iv: "18be159185426e086ae2d4368d93dda6"
server_key_version: "encryptionkey@0"
I20260620 11:33:16.177007 3654 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:33:16.177920 3669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:16.178164 3654 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:16.178248 3654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "e657dff0f27041e3a3a11dcf0225b72b"
format_stamp: "Formatted at 2026-06-20 11:33:16 on dist-test-slave-ctbw"
server_key: "9f89799f53602cea70a3617417959a08"
server_key_iv: "18be159185426e086ae2d4368d93dda6"
server_key_version: "encryptionkey@0"
I20260620 11:33:16.178304 3654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:16.201391 3654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:16.203873 3654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:16.204044 3654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:16.204284 3654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:16.204607 3654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:16.204660 3654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:16.204701 3654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:16.204730 3654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:16.210412 3654 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:41899
I20260620 11:33:16.210493 3782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:41899 every 8 connection(s)
I20260620 11:33:16.210810 3654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:33:16.214421 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 3654
I20260620 11:33:16.214493 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:33:16.214670 3022 external_mini_cluster.cc:1428] Setting key b5a353b5794a06c05a894b5e3dbfb022
Jun 20 11:33:16 dist-test-slave-ctbw krb5kdc[3547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955196, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:16.221174 3632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:16.211574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:53479 (local address 127.2.243.190:40683)
0620 11:33:16.211641 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:16.211643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:16.211850 (+ 207us) server_negotiation.cc:407] Connection header received
0620 11:33:16.212767 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:16.212769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:16.212779 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:16.212845 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:16.213505 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.214029 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.215414 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.215738 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.215816 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:16.218136 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:16.218137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:16.218138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:16.218145 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:16.219697 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:16.220125 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:16.220127 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:16.220127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:16.220195 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:16.220442 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:16.220443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:16.220443 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:16.220619 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:16.220683 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:16.220980 (+ 297us) server_negotiation.cc:299] Negotiation successful
0620 11:33:16.221084 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 11:33:16.221385 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:16.211553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40683 (local address 127.2.243.129:53479)
0620 11:33:16.211716 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:16.211767 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:16.212656 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:16.212913 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:16.212916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:16.213217 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:16.213403 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:16.213408 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.214143 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.214145 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:16.215307 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:16.215310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.215825 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.215826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:16.215926 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:16.216524 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:16.216536 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:16.218049 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:16.219799 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:16.219806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:16.219808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:16.220033 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:16.220260 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:16.220261 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:16.220262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:16.220372 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:16.220711 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:16.220716 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:16.220853 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 11:33:16.220959 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 11:33:16.221134 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":97,"threads_started":1}
I20260620 11:33:16.221838 3783 heartbeater.cc:347] Connected to a master server at 127.2.243.190:40683
I20260620 11:33:16.221944 3783 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:16.222150 3783 heartbeater.cc:511] Master 127.2.243.190:40683 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.243.130@KRBTEST.COM; defaulting to no policy
I20260620 11:33:16.222652 3592 ts_manager.cc:194] Registered new tserver with Master: e657dff0f27041e3a3a11dcf0225b72b (127.2.243.129:41899)
I20260620 11:33:16.223346 3592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:53479
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
I20260620 11:33:16.239436 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:40683
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:16.320263 3788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:16.320456 3788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:16.320479 3788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:16.321552 3788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:16.321591 3788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:16.321624 3788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:33:16.323114 3788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:40683
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:16.323308 3788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:16.323544 3788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:16.328311 3794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:16.328354 3793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:16.328474 3788 server_base.cc:1034] running on GCE node
W20260620 11:33:16.328354 3796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:16.328792 3788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:16.328985 3788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:16.330147 3788 hybrid_clock.cc:648] HybridClock initialized: now 1781955196330122 us; error 41 us; skew 500 ppm
Jun 20 11:33:16 dist-test-slave-ctbw krb5kdc[3547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955196, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:16.332656 3788 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:33:16.333132 3788 webserver.cc:460] Webserver started at http://127.2.243.130:41661/ using document root <none> and password file <none>
I20260620 11:33:16.333367 3788 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:16.333436 3788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:16.333600 3788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:16.334609 3788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "ac7f457f6c084acabe4b3ed97cab9638"
format_stamp: "Formatted at 2026-06-20 11:33:16 on dist-test-slave-ctbw"
server_key: "a77fdb25e88d3c25dffb21d1c58ae868"
server_key_iv: "24d272ff2f9f70e7fd50d5cbe9d109ad"
server_key_version: "encryptionkey@0"
I20260620 11:33:16.335019 3788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "ac7f457f6c084acabe4b3ed97cab9638"
format_stamp: "Formatted at 2026-06-20 11:33:16 on dist-test-slave-ctbw"
server_key: "a77fdb25e88d3c25dffb21d1c58ae868"
server_key_iv: "24d272ff2f9f70e7fd50d5cbe9d109ad"
server_key_version: "encryptionkey@0"
I20260620 11:33:16.336305 3788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:16.337020 3803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:16.337229 3788 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:33:16.337330 3788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "ac7f457f6c084acabe4b3ed97cab9638"
format_stamp: "Formatted at 2026-06-20 11:33:16 on dist-test-slave-ctbw"
server_key: "a77fdb25e88d3c25dffb21d1c58ae868"
server_key_iv: "24d272ff2f9f70e7fd50d5cbe9d109ad"
server_key_version: "encryptionkey@0"
I20260620 11:33:16.337409 3788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:16.358925 3788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:16.361114 3788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:16.361308 3788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:16.361548 3788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:16.361874 3788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:16.361928 3788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:16.361968 3788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:16.361999 3788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:16.367940 3788 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:39989
I20260620 11:33:16.367993 3916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:39989 every 8 connection(s)
I20260620 11:33:16.368324 3788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:33:16.374209 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 3788
I20260620 11:33:16.374295 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:33:16.374477 3022 external_mini_cluster.cc:1428] Setting key 8d55f10fc2a7160ff5d10bfbefa0c242
Jun 20 11:33:16 dist-test-slave-ctbw krb5kdc[3547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955196, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:16.378751 3632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:16.369189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:34055 (local address 127.2.243.190:40683)
0620 11:33:16.369264 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:16.369266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:16.369475 (+ 209us) server_negotiation.cc:407] Connection header received
0620 11:33:16.370472 (+ 997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:16.370473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:16.370483 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:16.370538 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:16.371188 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.371720 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.372785 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.372983 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.373049 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:16.375639 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:16.375640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:16.375641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:16.375648 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:16.377284 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:16.377816 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:16.377817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:16.377817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:16.377867 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:16.378156 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:16.378157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:16.378157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:16.378257 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:16.378310 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:16.378629 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 11:33:16.378689 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 11:33:16.379070 3919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:16.369170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40683 (local address 127.2.243.130:34055)
0620 11:33:16.369339 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:16.369373 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:16.370365 (+ 992us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:16.370605 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:16.370609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:16.370863 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:16.371078 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:16.371083 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.371825 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.371826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:16.372682 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:16.372685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.373082 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.373083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:16.373198 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:16.373826 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:16.373835 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:16.375548 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:16.377382 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:16.377389 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:16.377391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:16.377740 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:16.377932 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:16.377933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:16.377933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:16.378091 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:16.378323 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:16.378328 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:16.378490 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 11:33:16.378613 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 11:33:16.378792 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260620 11:33:16.379618 3917 heartbeater.cc:347] Connected to a master server at 127.2.243.190:40683
I20260620 11:33:16.379710 3917 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:16.379906 3917 heartbeater.cc:511] Master 127.2.243.190:40683 requested a full tablet report, sending...
I20260620 11:33:16.380296 3592 ts_manager.cc:194] Registered new tserver with Master: ac7f457f6c084acabe4b3ed97cab9638 (127.2.243.130:39989)
I20260620 11:33:16.380869 3592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:34055
WARNING: no policy specified for kudu/127.2.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
I20260620 11:33:16.399719 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:40683
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:37933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:16.480103 3922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:16.480345 3922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:16.480372 3922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:16.481559 3922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:16.481598 3922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:16.481624 3922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:33:16.483224 3922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:37933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:40683
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:16.483453 3922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:16.483681 3922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:16.488826 3927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:16.488858 3928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:16.488881 3922 server_base.cc:1034] running on GCE node
W20260620 11:33:16.488999 3930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:16.489257 3922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:16.489511 3922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:16.490710 3922 hybrid_clock.cc:648] HybridClock initialized: now 1781955196490680 us; error 45 us; skew 500 ppm
Jun 20 11:33:16 dist-test-slave-ctbw krb5kdc[3547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955196, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:16.493291 3922 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:33:16.493816 3922 webserver.cc:460] Webserver started at http://127.2.243.131:41773/ using document root <none> and password file <none>
I20260620 11:33:16.494057 3922 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:16.494133 3922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:16.494284 3922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:16.495404 3922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "169d28c409dd44369166b5ca7ddc4cc2"
format_stamp: "Formatted at 2026-06-20 11:33:16 on dist-test-slave-ctbw"
server_key: "b040f85b072913630287da90742c78e2"
server_key_iv: "092a5bb4a31efcf4e110bf7b9e750f5f"
server_key_version: "encryptionkey@0"
I20260620 11:33:16.495782 3922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "169d28c409dd44369166b5ca7ddc4cc2"
format_stamp: "Formatted at 2026-06-20 11:33:16 on dist-test-slave-ctbw"
server_key: "b040f85b072913630287da90742c78e2"
server_key_iv: "092a5bb4a31efcf4e110bf7b9e750f5f"
server_key_version: "encryptionkey@0"
I20260620 11:33:16.497080 3922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:33:16.497802 3937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:16.497996 3922 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:16.498093 3922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "169d28c409dd44369166b5ca7ddc4cc2"
format_stamp: "Formatted at 2026-06-20 11:33:16 on dist-test-slave-ctbw"
server_key: "b040f85b072913630287da90742c78e2"
server_key_iv: "092a5bb4a31efcf4e110bf7b9e750f5f"
server_key_version: "encryptionkey@0"
I20260620 11:33:16.498175 3922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:16.511051 3922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:16.512931 3922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:16.513087 3922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:16.513310 3922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:16.513603 3922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:16.513665 3922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:16.513706 3922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:16.513734 3922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:16.519383 3922 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:41059
I20260620 11:33:16.519424 4050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:41059 every 8 connection(s)
I20260620 11:33:16.519724 3922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:33:16.524168 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 3922
I20260620 11:33:16.524281 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:33:16.524442 3022 external_mini_cluster.cc:1428] Setting key 9a6ad2712d03394928adf0ba5e0652c8
Jun 20 11:33:16 dist-test-slave-ctbw krb5kdc[3547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955196, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:16.529048 3632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:16.520355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:50643 (local address 127.2.243.190:40683)
0620 11:33:16.520432 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:16.520434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:16.520635 (+ 201us) server_negotiation.cc:407] Connection header received
0620 11:33:16.521544 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:16.521545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:16.521555 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:16.521608 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:16.522164 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.522648 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.523676 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.523862 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.523927 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:16.526245 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:16.526247 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:16.526247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:16.526254 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:16.527570 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:16.528030 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:16.528031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:16.528032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:16.528129 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:16.528390 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:16.528391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:16.528391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:16.528504 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:16.528578 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:16.528877 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 11:33:16.528952 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 11:33:16.529237 4053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:16.520314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40683 (local address 127.2.243.131:50643)
0620 11:33:16.520490 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:16.520540 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:16.521455 (+ 915us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:16.521662 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:16.521665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:16.521914 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:16.522092 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:16.522096 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.522722 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.522723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:16.523591 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:16.523593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.523942 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.523942 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:16.524038 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:16.524610 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:16.524621 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:16.526159 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:16.527691 (+ 1532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:16.527713 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:16.527715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:16.527935 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:16.528199 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:16.528200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:16.528201 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:16.528311 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:16.528568 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:16.528573 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:16.528734 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 11:33:16.528856 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 11:33:16.529009 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
I20260620 11:33:16.529731 4051 heartbeater.cc:347] Connected to a master server at 127.2.243.190:40683
I20260620 11:33:16.529835 4051 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:16.529971 4051 heartbeater.cc:511] Master 127.2.243.190:40683 requested a full tablet report, sending...
I20260620 11:33:16.530324 3592 ts_manager.cc:194] Registered new tserver with Master: 169d28c409dd44369166b5ca7ddc4cc2 (127.2.243.131:41059)
I20260620 11:33:16.530711 3592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:50643
I20260620 11:33:16.536142 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:33:16.543748 3632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:16.537600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38196 (local address 127.2.243.190:40683)
0620 11:33:16.537653 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:16.537655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:16.537727 (+ 72us) server_negotiation.cc:407] Connection header received
0620 11:33:16.537767 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:16.537768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:16.537775 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:16.537825 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:16.538352 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.538822 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.539580 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.539767 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.539820 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:16.541193 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:16.541194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:16.541195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:16.541201 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:16.542598 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:16.542990 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:16.542991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:16.542991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:16.543041 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:16.543312 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:16.543312 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:16.543313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:16.543444 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:16.543517 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:16.543616 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 11:33:16.543672 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 11:33:16.546171 3592 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38196:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:33:16.547871 3592 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:33:16.554476 4065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:16.550271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33050 (local address 127.2.243.129:41899)
0620 11:33:16.550436 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:16.550442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:16.550648 (+ 206us) server_negotiation.cc:407] Connection header received
0620 11:33:16.550719 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:16.550721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:16.550737 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:16.550838 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:16.551000 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.551836 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.553398 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.554144 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.554234 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:16.554292 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:16.554348 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 11:33:16.554394 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":71,"threads_started":1}
I20260620 11:33:16.554492 4062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:16.550367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:39989 (local address 127.0.0.1:38680)
0620 11:33:16.550718 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:16.550726 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:16.550763 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:16.551156 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:16.551157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:16.551159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:16.551195 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:16.551196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.552492 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.552493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:16.553418 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:16.553420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.554259 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.554260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:16.554324 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:16.554331 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 11:33:16.554362 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 11:33:16.554399 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260620 11:33:16.554569 4064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:16.550237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36280 (local address 127.2.243.131:41059)
0620 11:33:16.550430 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:16.550436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:16.550613 (+ 177us) server_negotiation.cc:407] Connection header received
0620 11:33:16.550748 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:16.550751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:16.550768 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:16.550836 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:16.551081 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.552238 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.553481 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.553966 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.554066 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:16.554134 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:16.554438 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 11:33:16.554480 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":94,"threads_started":1}
I20260620 11:33:16.554677 4063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:16.550141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:41899 (local address 127.0.0.1:33050)
0620 11:33:16.550507 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:16.550549 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:16.550630 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:16.550843 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:16.550846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:16.550849 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:16.550937 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:16.550939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.551942 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.551943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:16.553323 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:16.553325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.554231 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.554232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:16.554284 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:16.554287 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:33:16.554337 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 11:33:16.554372 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"spinlock_wait_cycles":384,"thread_start_us":64,"threads_started":1}
I20260620 11:33:16.554690 4067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:16.550585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38680 (local address 127.2.243.130:39989)
0620 11:33:16.550905 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:16.550910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:16.550919 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:33:16.550954 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:16.550956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:16.550969 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:16.551043 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:16.551272 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.552389 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.553505 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.554170 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.554441 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:16.554502 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:16.554527 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 11:33:16.554591 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":72,"threads_started":1}
I20260620 11:33:16.554394 4066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:16.550141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:41059 (local address 127.0.0.1:36280)
0620 11:33:16.550507 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:16.550549 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:16.550630 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:16.550908 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:16.550909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:16.550911 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:16.550967 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:16.550969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.552357 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.552358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:16.553422 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:16.553424 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.554069 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.554070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:16.554158 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:16.554171 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 11:33:16.554248 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 11:33:16.554302 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":25,"spinlock_wait_cycles":768,"thread_start_us":86,"threads_started":1}
I20260620 11:33:16.555473 3985 tablet_service.cc:1467] Processing CreateTablet for tablet 4f70f715c1894be08b7a795e4af851f8 (DEFAULT_TABLE table=test-table [id=896ff43df9e749188cff4be1e0b46e8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:16.555523 3851 tablet_service.cc:1467] Processing CreateTablet for tablet 4f70f715c1894be08b7a795e4af851f8 (DEFAULT_TABLE table=test-table [id=896ff43df9e749188cff4be1e0b46e8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:16.555611 3717 tablet_service.cc:1467] Processing CreateTablet for tablet 4f70f715c1894be08b7a795e4af851f8 (DEFAULT_TABLE table=test-table [id=896ff43df9e749188cff4be1e0b46e8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:16.555835 3985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f70f715c1894be08b7a795e4af851f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:16.555860 3851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f70f715c1894be08b7a795e4af851f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:16.555872 3717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f70f715c1894be08b7a795e4af851f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:16.558247 4068 tablet_bootstrap.cc:492] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2: Bootstrap starting.
I20260620 11:33:16.558879 4068 tablet_bootstrap.cc:654] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:16.558982 4070 tablet_bootstrap.cc:492] T 4f70f715c1894be08b7a795e4af851f8 P e657dff0f27041e3a3a11dcf0225b72b: Bootstrap starting.
I20260620 11:33:16.559635 4070 tablet_bootstrap.cc:654] T 4f70f715c1894be08b7a795e4af851f8 P e657dff0f27041e3a3a11dcf0225b72b: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:16.559656 4068 log.cc:824] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:16.560259 4070 log.cc:824] T 4f70f715c1894be08b7a795e4af851f8 P e657dff0f27041e3a3a11dcf0225b72b: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:16.560506 4068 tablet_bootstrap.cc:492] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2: No bootstrap required, opened a new log
I20260620 11:33:16.560576 4068 ts_tablet_manager.cc:1397] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:33:16.560583 4069 tablet_bootstrap.cc:492] T 4f70f715c1894be08b7a795e4af851f8 P ac7f457f6c084acabe4b3ed97cab9638: Bootstrap starting.
I20260620 11:33:16.560928 4070 tablet_bootstrap.cc:492] T 4f70f715c1894be08b7a795e4af851f8 P e657dff0f27041e3a3a11dcf0225b72b: No bootstrap required, opened a new log
I20260620 11:33:16.561010 4070 ts_tablet_manager.cc:1397] T 4f70f715c1894be08b7a795e4af851f8 P e657dff0f27041e3a3a11dcf0225b72b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 11:33:16.561609 4069 tablet_bootstrap.cc:654] T 4f70f715c1894be08b7a795e4af851f8 P ac7f457f6c084acabe4b3ed97cab9638: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:16.562134 4069 log.cc:824] T 4f70f715c1894be08b7a795e4af851f8 P ac7f457f6c084acabe4b3ed97cab9638: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:16.562917 4069 tablet_bootstrap.cc:492] T 4f70f715c1894be08b7a795e4af851f8 P ac7f457f6c084acabe4b3ed97cab9638: No bootstrap required, opened a new log
I20260620 11:33:16.562986 4069 ts_tablet_manager.cc:1397] T 4f70f715c1894be08b7a795e4af851f8 P ac7f457f6c084acabe4b3ed97cab9638: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:33:16.562927 4070 raft_consensus.cc:348] T 4f70f715c1894be08b7a795e4af851f8 P e657dff0f27041e3a3a11dcf0225b72b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e657dff0f27041e3a3a11dcf0225b72b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 41899 } } peers { permanent_uuid: "169d28c409dd44369166b5ca7ddc4cc2" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 41059 } } peers { permanent_uuid: "ac7f457f6c084acabe4b3ed97cab9638" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39989 } }
I20260620 11:33:16.563141 4070 raft_consensus.cc:374] T 4f70f715c1894be08b7a795e4af851f8 P e657dff0f27041e3a3a11dcf0225b72b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:16.563185 4070 raft_consensus.cc:729] T 4f70f715c1894be08b7a795e4af851f8 P e657dff0f27041e3a3a11dcf0225b72b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e657dff0f27041e3a3a11dcf0225b72b, State: Initialized, Role: FOLLOWER
I20260620 11:33:16.563091 4068 raft_consensus.cc:348] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e657dff0f27041e3a3a11dcf0225b72b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 41899 } } peers { permanent_uuid: "169d28c409dd44369166b5ca7ddc4cc2" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 41059 } } peers { permanent_uuid: "ac7f457f6c084acabe4b3ed97cab9638" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39989 } }
I20260620 11:33:16.563267 4068 raft_consensus.cc:374] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:16.563311 4068 raft_consensus.cc:729] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 169d28c409dd44369166b5ca7ddc4cc2, State: Initialized, Role: FOLLOWER
I20260620 11:33:16.563319 4070 consensus_queue.cc:260] T 4f70f715c1894be08b7a795e4af851f8 P e657dff0f27041e3a3a11dcf0225b72b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e657dff0f27041e3a3a11dcf0225b72b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 41899 } } peers { permanent_uuid: "169d28c409dd44369166b5ca7ddc4cc2" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 41059 } } peers { permanent_uuid: "ac7f457f6c084acabe4b3ed97cab9638" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39989 } }
I20260620 11:33:16.563422 4068 consensus_queue.cc:260] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e657dff0f27041e3a3a11dcf0225b72b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 41899 } } peers { permanent_uuid: "169d28c409dd44369166b5ca7ddc4cc2" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 41059 } } peers { permanent_uuid: "ac7f457f6c084acabe4b3ed97cab9638" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39989 } }
I20260620 11:33:16.563591 4070 ts_tablet_manager.cc:1428] T 4f70f715c1894be08b7a795e4af851f8 P e657dff0f27041e3a3a11dcf0225b72b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 11:33:16.563678 4068 ts_tablet_manager.cc:1428] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:33:16.563720 3783 heartbeater.cc:503] Master 127.2.243.190:40683 was elected leader, sending a full tablet report...
I20260620 11:33:16.563948 4051 heartbeater.cc:503] Master 127.2.243.190:40683 was elected leader, sending a full tablet report...
I20260620 11:33:16.565264 4069 raft_consensus.cc:348] T 4f70f715c1894be08b7a795e4af851f8 P ac7f457f6c084acabe4b3ed97cab9638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e657dff0f27041e3a3a11dcf0225b72b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 41899 } } peers { permanent_uuid: "169d28c409dd44369166b5ca7ddc4cc2" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 41059 } } peers { permanent_uuid: "ac7f457f6c084acabe4b3ed97cab9638" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39989 } }
I20260620 11:33:16.565439 4069 raft_consensus.cc:374] T 4f70f715c1894be08b7a795e4af851f8 P ac7f457f6c084acabe4b3ed97cab9638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:16.565487 4069 raft_consensus.cc:729] T 4f70f715c1894be08b7a795e4af851f8 P ac7f457f6c084acabe4b3ed97cab9638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac7f457f6c084acabe4b3ed97cab9638, State: Initialized, Role: FOLLOWER
I20260620 11:33:16.565598 4069 consensus_queue.cc:260] T 4f70f715c1894be08b7a795e4af851f8 P ac7f457f6c084acabe4b3ed97cab9638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e657dff0f27041e3a3a11dcf0225b72b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 41899 } } peers { permanent_uuid: "169d28c409dd44369166b5ca7ddc4cc2" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 41059 } } peers { permanent_uuid: "ac7f457f6c084acabe4b3ed97cab9638" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39989 } }
I20260620 11:33:16.565868 4069 ts_tablet_manager.cc:1428] T 4f70f715c1894be08b7a795e4af851f8 P ac7f457f6c084acabe4b3ed97cab9638: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 11:33:16.565917 3917 heartbeater.cc:503] Master 127.2.243.190:40683 was elected leader, sending a full tablet report...
W20260620 11:33:16.618919 3918 tablet.cc:2369] T 4f70f715c1894be08b7a795e4af851f8 P ac7f457f6c084acabe4b3ed97cab9638: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 11:33:16.711597 3784 tablet.cc:2369] T 4f70f715c1894be08b7a795e4af851f8 P e657dff0f27041e3a3a11dcf0225b72b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:33:16.762331 4075 raft_consensus.cc:482] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:33:16.762472 4075 raft_consensus.cc:504] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e657dff0f27041e3a3a11dcf0225b72b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 41899 } } peers { permanent_uuid: "169d28c409dd44369166b5ca7ddc4cc2" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 41059 } } peers { permanent_uuid: "ac7f457f6c084acabe4b3ed97cab9638" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39989 } }
I20260620 11:33:16.762791 4075 leader_election.cc:288] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e657dff0f27041e3a3a11dcf0225b72b (127.2.243.129:41899), ac7f457f6c084acabe4b3ed97cab9638 (127.2.243.130:39989)
I20260620 11:33:16.765916 4065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:16.763145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:59653 (local address 127.2.243.129:41899)
0620 11:33:16.763212 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:16.763215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:16.763255 (+ 40us) server_negotiation.cc:407] Connection header received
0620 11:33:16.763332 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:16.763333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:16.763353 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:16.763465 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:16.763632 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.764122 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.765105 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.765532 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.765641 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:16.765651 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:16.765810 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 11:33:16.765844 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 11:33:16.765916 4053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:16.763073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:41899 (local address 127.2.243.131:59653)
0620 11:33:16.763184 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:16.763198 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:16.763266 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:16.763470 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:16.763472 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:16.763474 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:16.763549 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:16.763551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.764214 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.764215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:16.764988 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:16.764990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.765644 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.765645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:16.765731 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:16.765734 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:33:16.765805 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 11:33:16.765844 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260620 11:33:16.766260 4067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:16.763288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:57279 (local address 127.2.243.130:39989)
0620 11:33:16.763343 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:16.763345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:16.763470 (+ 125us) server_negotiation.cc:407] Connection header received
0620 11:33:16.763532 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:16.763533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:16.763554 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:16.763636 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:16.763807 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.764298 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.765427 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.765839 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.765943 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:16.765954 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:16.766150 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 11:33:16.766188 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 11:33:16.766428 3737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f70f715c1894be08b7a795e4af851f8" candidate_uuid: "169d28c409dd44369166b5ca7ddc4cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e657dff0f27041e3a3a11dcf0225b72b" is_pre_election: true
I20260620 11:33:16.766505 4077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:16.763227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:39989 (local address 127.2.243.131:57279)
0620 11:33:16.763384 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:16.763393 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:16.763473 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:16.763642 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:16.763643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:16.763645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:16.763728 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:16.763730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.764391 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.764392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:16.765332 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:16.765334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.765959 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.765960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:16.766039 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:16.766044 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:33:16.766363 (+ 319us) client_negotiation.cc:240] Negotiation successful
0620 11:33:16.766408 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":118,"threads_started":1}
I20260620 11:33:16.766626 3737 raft_consensus.cc:2454] T 4f70f715c1894be08b7a795e4af851f8 P e657dff0f27041e3a3a11dcf0225b72b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 169d28c409dd44369166b5ca7ddc4cc2 in term 0.
I20260620 11:33:16.766880 3938 leader_election.cc:302] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 169d28c409dd44369166b5ca7ddc4cc2, e657dff0f27041e3a3a11dcf0225b72b; no voters:
I20260620 11:33:16.767007 3871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f70f715c1894be08b7a795e4af851f8" candidate_uuid: "169d28c409dd44369166b5ca7ddc4cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac7f457f6c084acabe4b3ed97cab9638" is_pre_election: true
I20260620 11:33:16.767060 4075 raft_consensus.cc:2781] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:33:16.767112 4075 raft_consensus.cc:482] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:33:16.767136 4075 raft_consensus.cc:3037] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:16.767172 3871 raft_consensus.cc:2454] T 4f70f715c1894be08b7a795e4af851f8 P ac7f457f6c084acabe4b3ed97cab9638 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 169d28c409dd44369166b5ca7ddc4cc2 in term 0.
I20260620 11:33:16.767875 4075 raft_consensus.cc:504] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e657dff0f27041e3a3a11dcf0225b72b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 41899 } } peers { permanent_uuid: "169d28c409dd44369166b5ca7ddc4cc2" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 41059 } } peers { permanent_uuid: "ac7f457f6c084acabe4b3ed97cab9638" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39989 } }
I20260620 11:33:16.768021 4075 leader_election.cc:288] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2 [CANDIDATE]: Term 1 election: Requested vote from peers e657dff0f27041e3a3a11dcf0225b72b (127.2.243.129:41899), ac7f457f6c084acabe4b3ed97cab9638 (127.2.243.130:39989)
I20260620 11:33:16.768203 3737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f70f715c1894be08b7a795e4af851f8" candidate_uuid: "169d28c409dd44369166b5ca7ddc4cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e657dff0f27041e3a3a11dcf0225b72b"
I20260620 11:33:16.768268 3737 raft_consensus.cc:3037] T 4f70f715c1894be08b7a795e4af851f8 P e657dff0f27041e3a3a11dcf0225b72b [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:16.768901 3737 raft_consensus.cc:2454] T 4f70f715c1894be08b7a795e4af851f8 P e657dff0f27041e3a3a11dcf0225b72b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 169d28c409dd44369166b5ca7ddc4cc2 in term 1.
I20260620 11:33:16.769101 3871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f70f715c1894be08b7a795e4af851f8" candidate_uuid: "169d28c409dd44369166b5ca7ddc4cc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac7f457f6c084acabe4b3ed97cab9638"
I20260620 11:33:16.769112 3938 leader_election.cc:302] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 169d28c409dd44369166b5ca7ddc4cc2, e657dff0f27041e3a3a11dcf0225b72b; no voters:
I20260620 11:33:16.769172 3871 raft_consensus.cc:3037] T 4f70f715c1894be08b7a795e4af851f8 P ac7f457f6c084acabe4b3ed97cab9638 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:16.769296 4075 raft_consensus.cc:2781] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:16.769469 4075 raft_consensus.cc:686] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2 [term 1 LEADER]: Becoming Leader. State: Replica: 169d28c409dd44369166b5ca7ddc4cc2, State: Running, Role: LEADER
I20260620 11:33:16.769574 4075 consensus_queue.cc:237] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e657dff0f27041e3a3a11dcf0225b72b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 41899 } } peers { permanent_uuid: "169d28c409dd44369166b5ca7ddc4cc2" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 41059 } } peers { permanent_uuid: "ac7f457f6c084acabe4b3ed97cab9638" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39989 } }
I20260620 11:33:16.770162 3871 raft_consensus.cc:2454] T 4f70f715c1894be08b7a795e4af851f8 P ac7f457f6c084acabe4b3ed97cab9638 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 169d28c409dd44369166b5ca7ddc4cc2 in term 1.
W20260620 11:33:16.770186 4052 tablet.cc:2369] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:33:16.770592 3592 catalog_manager.cc:5495] T 4f70f715c1894be08b7a795e4af851f8 P 169d28c409dd44369166b5ca7ddc4cc2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 169d28c409dd44369166b5ca7ddc4cc2 (127.2.243.131). New cstate: current_term: 1 leader_uuid: "169d28c409dd44369166b5ca7ddc4cc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e657dff0f27041e3a3a11dcf0225b72b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "169d28c409dd44369166b5ca7ddc4cc2" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 41059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac7f457f6c084acabe4b3ed97cab9638" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39989 } health_report { overall_health: UNKNOWN } } }
Jun 20 11:33:16 dist-test-slave-ctbw krb5kdc[3547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.129@KRBTEST.COM
I20260620 11:33:16.789265 4065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:16.781460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33052 (local address 127.2.243.129:41899)
0620 11:33:16.781523 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:16.781525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:16.781679 (+ 154us) server_negotiation.cc:407] Connection header received
0620 11:33:16.781752 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:16.781753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:16.781764 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:16.781838 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:16.782465 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.783006 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.783732 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.783940 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.784026 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:16.786393 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:16.786404 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:16.786404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:16.786411 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:16.788072 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:16.788447 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:16.788451 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:16.788452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:16.788518 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:16.788728 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:16.788730 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:16.788730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:16.788877 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:16.789031 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:16.789074 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 11:33:16.789158 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 11:33:16 dist-test-slave-ctbw krb5kdc[3547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955196, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 11:33:16 dist-test-slave-ctbw krb5kdc[3547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955196, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.243.129@KRBTEST.COM
I20260620 11:33:16.815110 4065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:16.807788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33062 (local address 127.2.243.129:41899)
0620 11:33:16.807858 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:16.807860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:16.807975 (+ 115us) server_negotiation.cc:407] Connection header received
0620 11:33:16.808065 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:16.808066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:16.808082 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:16.808150 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:16.808724 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.809190 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.809920 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.810107 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.810193 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:16.812543 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:16.812545 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:16.812545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:16.812551 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:16.813991 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:16.814291 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:16.814292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:16.814292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:16.814340 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:16.814508 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:16.814508 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:16.814508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:16.814590 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:16.814680 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:16.814965 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 11:33:16.815027 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 11:33:16 dist-test-slave-ctbw krb5kdc[3547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 11:33:16 dist-test-slave-ctbw krb5kdc[3547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955196, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.129@KRBTEST.COM
I20260620 11:33:16.838186 4065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:16.830777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33072 (local address 127.2.243.129:41899)
0620 11:33:16.830844 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:16.830847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:16.830960 (+ 113us) server_negotiation.cc:407] Connection header received
0620 11:33:16.831046 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:16.831047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:16.831057 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:16.831129 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:16.831714 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.832199 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.832937 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:16.833130 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:16.833216 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:16.835558 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:16.835559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:16.835559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:16.835568 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:16.837056 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:16.837406 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:16.837407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:16.837408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:16.837464 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:16.837664 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:16.837665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:16.837666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:16.837796 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:16.837901 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:16.838005 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 11:33:16.838084 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 11:33:16.838757 3697 tablet_service.cc:2908] Rejecting scan request for tablet 4f70f715c1894be08b7a795e4af851f8: Uninitialized: clean time has not yet been initialized
I20260620 11:33:16.839623 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 3654
I20260620 11:33:16.844694 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 3788
I20260620 11:33:16.849138 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 3922
I20260620 11:33:16.853426 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 3560
2026-06-20T11:33:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2136 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:33:16 dist-test-slave-ctbw krb5kdc[4092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:33:16 dist-test-slave-ctbw krb5kdc[4092](info): set up 2 sockets
Jun 20 11:33:16 dist-test-slave-ctbw krb5kdc[4092](info): commencing operation
krb5kdc: starting...
W20260620 11:33:17.914839 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:33:17 dist-test-slave-ctbw krb5kdc[4092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:33:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:33:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:33:18.034073 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:41559
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:44207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:41559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:18.115674 4105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:18.115890 4105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:18.115912 4105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:18.117022 4105 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:33:18.117055 4105 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 11:33:18.117069 4105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:18.117081 4105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:18.117093 4105 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:33:18.117106 4105 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:33:18.118492 4105 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:44207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:41559
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:41559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:18.118685 4105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:18.118853 4105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:18.123978 4111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:18.123960 4113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:18.123960 4110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:18.124076 4105 server_base.cc:1034] running on GCE node
I20260620 11:33:18.124504 4105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:18.124809 4105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:18.125967 4105 hybrid_clock.cc:648] HybridClock initialized: now 1781955198125953 us; error 31 us; skew 500 ppm
Jun 20 11:33:18 dist-test-slave-ctbw krb5kdc[4092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955198, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:18.128623 4105 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:33:18.129129 4105 webserver.cc:460] Webserver started at http://127.2.243.190:39243/ using document root <none> and password file <none>
I20260620 11:33:18.129370 4105 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:18.129441 4105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:18.129594 4105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:18.130648 4105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "9c3d7bf740d24cf2bedac5c416dd80b2"
format_stamp: "Formatted at 2026-06-20 11:33:18 on dist-test-slave-ctbw"
server_key: "0dfaafa3adf2e553f8ef98a2f9035f07"
server_key_iv: "5e9bc4bba6619d1788d54507c8bc51a5"
server_key_version: "encryptionkey@0"
I20260620 11:33:18.131047 4105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "9c3d7bf740d24cf2bedac5c416dd80b2"
format_stamp: "Formatted at 2026-06-20 11:33:18 on dist-test-slave-ctbw"
server_key: "0dfaafa3adf2e553f8ef98a2f9035f07"
server_key_iv: "5e9bc4bba6619d1788d54507c8bc51a5"
server_key_version: "encryptionkey@0"
I20260620 11:33:18.132519 4105 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:18.133304 4120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:18.133499 4105 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:18.133602 4105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "9c3d7bf740d24cf2bedac5c416dd80b2"
format_stamp: "Formatted at 2026-06-20 11:33:18 on dist-test-slave-ctbw"
server_key: "0dfaafa3adf2e553f8ef98a2f9035f07"
server_key_iv: "5e9bc4bba6619d1788d54507c8bc51a5"
server_key_version: "encryptionkey@0"
I20260620 11:33:18.133685 4105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:18.147696 4105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:18.150104 4105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:18.150271 4105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:18.155177 4105 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:41559
I20260620 11:33:18.155232 4182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:41559 every 8 connection(s)
I20260620 11:33:18.155566 4105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:33:18.156206 4183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:18.158802 4183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2: Bootstrap starting.
I20260620 11:33:18.159097 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 4105
I20260620 11:33:18.159173 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:33:18.159345 3022 external_mini_cluster.cc:1428] Setting key 27d0858987d8cf79d2c5b288d329752d
I20260620 11:33:18.159611 4183 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:18.160009 4183 log.cc:824] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:18.160818 4183 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2: No bootstrap required, opened a new log
I20260620 11:33:18.162838 4183 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3d7bf740d24cf2bedac5c416dd80b2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41559 } }
I20260620 11:33:18.163053 4183 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:18.163082 4183 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c3d7bf740d24cf2bedac5c416dd80b2, State: Initialized, Role: FOLLOWER
I20260620 11:33:18.163235 4183 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3d7bf740d24cf2bedac5c416dd80b2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41559 } }
I20260620 11:33:18.163312 4183 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:33:18.163360 4183 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:33:18.163414 4183 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:18.164078 4183 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3d7bf740d24cf2bedac5c416dd80b2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41559 } }
I20260620 11:33:18.164204 4183 leader_election.cc:302] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c3d7bf740d24cf2bedac5c416dd80b2; no voters:
I20260620 11:33:18.164412 4183 leader_election.cc:288] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:33:18.164529 4188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:18.164703 4183 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:18.165707 4188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2 [term 1 LEADER]: Becoming Leader. State: Replica: 9c3d7bf740d24cf2bedac5c416dd80b2, State: Running, Role: LEADER
I20260620 11:33:18.165807 4188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3d7bf740d24cf2bedac5c416dd80b2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41559 } }
Jun 20 11:33:18 dist-test-slave-ctbw krb5kdc[4092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:18.166499 4190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c3d7bf740d24cf2bedac5c416dd80b2. Latest consensus state: current_term: 1 leader_uuid: "9c3d7bf740d24cf2bedac5c416dd80b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3d7bf740d24cf2bedac5c416dd80b2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41559 } } }
I20260620 11:33:18.166584 4190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2 [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:18.167039 4198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:33:18.167527 4189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c3d7bf740d24cf2bedac5c416dd80b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c3d7bf740d24cf2bedac5c416dd80b2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41559 } } }
I20260620 11:33:18.167606 4189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2 [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:18.167671 4198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:33:18.169302 4198 catalog_manager.cc:1347] Generated new cluster ID: bffa4508263e4b94954fba8651370a2b
I20260620 11:33:18.169358 4198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:33:18.170737 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:18.159943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46960 (local address 127.2.243.190:41559)
0620 11:33:18.160103 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:18.160114 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:18.160148 (+ 34us) server_negotiation.cc:407] Connection header received
0620 11:33:18.160729 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:18.160739 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:18.160805 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:18.160974 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:18.161620 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.162510 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.163542 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.163955 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.164078 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:18.166930 (+ 2852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:18.166935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:18.166935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:18.166947 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:18.169057 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:18.169457 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:18.169460 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:18.169460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:18.169560 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:18.169807 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:18.169808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:18.169809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:18.170114 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:18.170261 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:18.170387 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 11:33:18.170521 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260620 11:33:18.177418 4198 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:33:18.177942 4198 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:33:18.183576 4198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2: Generated new TSK 0
I20260620 11:33:18.183779 4198 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
I20260620 11:33:18.209766 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:41559
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:44207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:18.292816 4209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:18.293061 4209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:18.293097 4209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:18.294250 4209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:18.294322 4209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:18.294355 4209 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 11:33:18.294395 4209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:33:18.295840 4209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:44207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.243.190:41559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:18.296118 4209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:18.296350 4209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:18.301203 4217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:18.301258 4215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:18.301191 4214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:18.301402 4209 server_base.cc:1034] running on GCE node
I20260620 11:33:18.301616 4209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:18.301817 4209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:18.302930 4209 hybrid_clock.cc:648] HybridClock initialized: now 1781955198302909 us; error 31 us; skew 500 ppm
Jun 20 11:33:18 dist-test-slave-ctbw krb5kdc[4092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955198, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:18.305487 4209 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:33:18.305919 4209 webserver.cc:460] Webserver started at http://127.2.243.129:45581/ using document root <none> and password file <none>
I20260620 11:33:18.306156 4209 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:18.306200 4209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:18.306357 4209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:18.307469 4209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1"
format_stamp: "Formatted at 2026-06-20 11:33:18 on dist-test-slave-ctbw"
server_key: "35d92474152ca74d89a9a809cde03485"
server_key_iv: "7baa2e35aed135f1a7658501c188771e"
server_key_version: "encryptionkey@0"
I20260620 11:33:18.307825 4209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1"
format_stamp: "Formatted at 2026-06-20 11:33:18 on dist-test-slave-ctbw"
server_key: "35d92474152ca74d89a9a809cde03485"
server_key_iv: "7baa2e35aed135f1a7658501c188771e"
server_key_version: "encryptionkey@0"
I20260620 11:33:18.309098 4209 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:33:18.309856 4224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:18.310032 4209 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:18.310129 4209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1"
format_stamp: "Formatted at 2026-06-20 11:33:18 on dist-test-slave-ctbw"
server_key: "35d92474152ca74d89a9a809cde03485"
server_key_iv: "7baa2e35aed135f1a7658501c188771e"
server_key_version: "encryptionkey@0"
I20260620 11:33:18.310209 4209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:18.322270 4209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:18.324539 4209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:18.324724 4209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:18.325289 4209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:18.325362 4209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:18.325412 4209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:18.325445 4209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 11:33:18 dist-test-slave-ctbw krb5kdc[4092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955198, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:18.333002 4209 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:40897
I20260620 11:33:18.333081 4339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:40897 every 8 connection(s)
I20260620 11:33:18.333398 4209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:33:18.335175 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 4209
I20260620 11:33:18.335291 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:33:18.335520 3022 external_mini_cluster.cc:1428] Setting key 1ff30e5e3f068d67a3838223e7ca1eaf
I20260620 11:33:18.336407 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:18.325810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:48987 (local address 127.2.243.190:41559)
0620 11:33:18.325886 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:18.325888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:18.326052 (+ 164us) server_negotiation.cc:407] Connection header received
0620 11:33:18.326562 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:18.326564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:18.326580 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:18.326680 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:18.327344 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.328153 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.329753 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.330145 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.330245 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:18.333153 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:18.333155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:18.333155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:18.333163 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:18.334696 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:18.335231 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:18.335233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:18.335233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:18.335307 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:18.335624 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:18.335626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:18.335626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:18.335795 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:18.335861 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:18.336190 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 11:33:18.336286 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 11:33:18.336585 4233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:18.325768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41559 (local address 127.2.243.129:48987)
0620 11:33:18.325921 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:18.325960 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:18.326462 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:18.326728 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:18.326731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:18.327045 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:18.327232 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:18.327237 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.328245 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.328246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:18.329644 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:18.329647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.330275 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.330276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:18.330456 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:18.331047 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:18.331058 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:18.333034 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:18.334830 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:18.334836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:18.334838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:18.335147 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:18.335396 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:18.335397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:18.335398 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:18.335535 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:18.335867 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:18.335875 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:18.336012 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 11:33:18.336132 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 11:33:18.336319 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260620 11:33:18.337791 4340 heartbeater.cc:347] Connected to a master server at 127.2.243.190:41559
I20260620 11:33:18.337874 4340 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:18.338043 4340 heartbeater.cc:511] Master 127.2.243.190:41559 requested a full tablet report, sending...
I20260620 11:33:18.338512 4136 ts_manager.cc:194] Registered new tserver with Master: dd7cab7e392f4ab2bf49e6f936c1ace1 (127.2.243.129:40897)
I20260620 11:33:18.339493 4136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:48987
WARNING: no policy specified for kudu/127.2.243.130@KRBTEST.COM; defaulting to no policy
I20260620 11:33:18.344712 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:18.338201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:52831 (local address 127.2.243.190:41559)
0620 11:33:18.338241 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:18.338244 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:18.339042 (+ 798us) server_negotiation.cc:407] Connection header received
0620 11:33:18.339114 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:18.339115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:18.339125 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:18.339200 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:18.339528 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.340032 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.340749 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.340957 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.341044 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:18.342033 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:18.342034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:18.342034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:18.342041 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:18.343524 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:18.343923 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:18.343924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:18.343925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:18.343984 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:18.344187 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:18.344188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:18.344188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:18.344307 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:18.344383 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:18.344542 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 11:33:18.344615 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:33:18.344723 4347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:18.338207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41559 (local address 127.2.243.129:52831)
0620 11:33:18.338961 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:18.338973 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:18.339055 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:18.339205 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:18.339206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:18.339390 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:18.339441 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:18.339442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.340124 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.340125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:18.340662 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:18.340664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.341045 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.341046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:18.341112 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:18.341618 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:18.341624 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:18.341920 (+ 296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:18.343626 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:18.343627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:18.343628 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:18.343835 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:18.344050 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:18.344051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:18.344051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:18.344110 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:18.344373 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:18.344374 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:18.344425 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 11:33:18.344516 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 11:33:18.344617 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":82,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
I20260620 11:33:18.361418 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:41559
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:44207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:18.443584 4349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:18.443776 4349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:18.443800 4349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:18.444903 4349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:18.444939 4349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:18.444960 4349 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 11:33:18.444977 4349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:33:18.446439 4349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:44207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.243.190:41559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:18.446630 4349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:18.446803 4349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:18.451786 4354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:18.451795 4357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:18.451831 4355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:18.452261 4349 server_base.cc:1034] running on GCE node
I20260620 11:33:18.452551 4349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:18.452838 4349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:18.454035 4349 hybrid_clock.cc:648] HybridClock initialized: now 1781955198454028 us; error 38 us; skew 500 ppm
Jun 20 11:33:18 dist-test-slave-ctbw krb5kdc[4092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955198, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:18.456779 4349 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:33:18.457278 4349 webserver.cc:460] Webserver started at http://127.2.243.130:33085/ using document root <none> and password file <none>
I20260620 11:33:18.457526 4349 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:18.457597 4349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:18.457741 4349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:18.458817 4349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "d51f2981b6c34325adc2d0191d45f729"
format_stamp: "Formatted at 2026-06-20 11:33:18 on dist-test-slave-ctbw"
server_key: "9d7b7d28c722079fd3d2552b6d72839f"
server_key_iv: "48e810b23f5d8a38aabe67b228448c0d"
server_key_version: "encryptionkey@0"
I20260620 11:33:18.459249 4349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "d51f2981b6c34325adc2d0191d45f729"
format_stamp: "Formatted at 2026-06-20 11:33:18 on dist-test-slave-ctbw"
server_key: "9d7b7d28c722079fd3d2552b6d72839f"
server_key_iv: "48e810b23f5d8a38aabe67b228448c0d"
server_key_version: "encryptionkey@0"
I20260620 11:33:18.460592 4349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:18.461325 4364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:18.461524 4349 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:18.461622 4349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "d51f2981b6c34325adc2d0191d45f729"
format_stamp: "Formatted at 2026-06-20 11:33:18 on dist-test-slave-ctbw"
server_key: "9d7b7d28c722079fd3d2552b6d72839f"
server_key_iv: "48e810b23f5d8a38aabe67b228448c0d"
server_key_version: "encryptionkey@0"
I20260620 11:33:18.461704 4349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:18.471728 4349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:18.473649 4349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:18.473822 4349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:18.474413 4349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:18.474462 4349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:18.474498 4349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:18.474516 4349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 11:33:18 dist-test-slave-ctbw krb5kdc[4092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955198, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:18.481549 4349 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:37443
I20260620 11:33:18.481621 4479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:37443 every 8 connection(s)
I20260620 11:33:18.481914 4349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:33:18.483943 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:18.474784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:35465 (local address 127.2.243.190:41559)
0620 11:33:18.474846 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:18.474848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:18.475103 (+ 255us) server_negotiation.cc:407] Connection header received
0620 11:33:18.475653 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:18.475654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:18.475667 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:18.475718 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:18.476462 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.476979 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.477999 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.478237 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.478314 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:18.481138 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:18.481139 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:18.481139 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:18.481146 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:18.482692 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:18.483167 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:18.483168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:18.483168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:18.483219 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:18.483440 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:18.483440 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:18.483440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:18.483540 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:18.483587 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:18.483822 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 11:33:18.483881 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 11:33:18.484115 4373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:18.474788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41559 (local address 127.2.243.130:35465)
0620 11:33:18.474967 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:18.475006 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:18.475534 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:18.475789 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:18.475792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:18.476076 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:18.476363 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:18.476369 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.477070 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.477071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:18.477894 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:18.477897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.478326 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.478327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:18.478472 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:18.479043 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:18.479052 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:18.481046 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:18.482765 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:18.482769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:18.482771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:18.483099 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:18.483280 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:18.483281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:18.483281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:18.483374 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:18.483593 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:18.483597 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:18.483705 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 11:33:18.483779 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 11:33:18.483920 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":100,"threads_started":1}
I20260620 11:33:18.484582 4480 heartbeater.cc:347] Connected to a master server at 127.2.243.190:41559
I20260620 11:33:18.484668 4480 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:18.484855 4480 heartbeater.cc:511] Master 127.2.243.190:41559 requested a full tablet report, sending...
I20260620 11:33:18.485216 4136 ts_manager.cc:194] Registered new tserver with Master: d51f2981b6c34325adc2d0191d45f729 (127.2.243.130:37443)
I20260620 11:33:18.485714 4136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:35465
I20260620 11:33:18.486467 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 4349
I20260620 11:33:18.486569 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:33:18.486747 3022 external_mini_cluster.cc:1428] Setting key b7515702ed082db5f9f87f014758a9b5
I20260620 11:33:18.491942 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:18.485664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:49719 (local address 127.2.243.190:41559)
0620 11:33:18.485722 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:18.485724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:18.485798 (+ 74us) server_negotiation.cc:407] Connection header received
0620 11:33:18.485847 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:18.485848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:18.485858 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:18.485929 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:18.486417 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.487146 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.487914 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.488220 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.488294 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:18.489394 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:18.489395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:18.489395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:18.489402 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:18.490757 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:18.491191 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:18.491192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:18.491192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:18.491248 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:18.491470 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:18.491471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:18.491471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:18.491603 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:18.491657 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:18.491824 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 11:33:18.491880 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 11:33:18.492250 4486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:18.485608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41559 (local address 127.2.243.130:49719)
0620 11:33:18.485722 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:18.485734 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:18.485792 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:18.485942 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:18.485943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:18.486164 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:18.486232 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:18.486234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.487237 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.487238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:18.487830 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:18.487832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.488336 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.488337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:18.488391 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:18.488860 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:18.488866 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:18.489286 (+ 420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:18.490857 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:18.490858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:18.490859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:18.491111 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:18.491328 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:18.491329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:18.491329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:18.491403 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:18.491669 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:18.491670 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:18.491720 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 11:33:18.491807 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 11:33:18.491926 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":65,"threads_started":1}
WARNING: no policy specified for kudu/127.2.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
I20260620 11:33:18.513849 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:41559
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:44207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:18.598979 4489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:18.599169 4489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:18.599192 4489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:18.600274 4489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:18.600307 4489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:18.600327 4489 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 11:33:18.600343 4489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:33:18.601655 4489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:44207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.243.190:41559
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:18.601826 4489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:18.601995 4489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:18.606853 4497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:18.606958 4494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:18.606921 4495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:18.607026 4489 server_base.cc:1034] running on GCE node
I20260620 11:33:18.607317 4489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:18.607542 4489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:18.608682 4489 hybrid_clock.cc:648] HybridClock initialized: now 1781955198608660 us; error 37 us; skew 500 ppm
Jun 20 11:33:18 dist-test-slave-ctbw krb5kdc[4092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955198, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:18.611141 4489 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:33:18.611598 4489 webserver.cc:460] Webserver started at http://127.2.243.131:39937/ using document root <none> and password file <none>
I20260620 11:33:18.611821 4489 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:18.611868 4489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:18.612041 4489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:18.613161 4489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "e6987b754865402bbf0d19c7bbe9c1e9"
format_stamp: "Formatted at 2026-06-20 11:33:18 on dist-test-slave-ctbw"
server_key: "24ba1cb8248796ed9b2dfc15732474de"
server_key_iv: "e04ee008acb46fee78ab4aa97a7779fb"
server_key_version: "encryptionkey@0"
I20260620 11:33:18.613519 4489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "e6987b754865402bbf0d19c7bbe9c1e9"
format_stamp: "Formatted at 2026-06-20 11:33:18 on dist-test-slave-ctbw"
server_key: "24ba1cb8248796ed9b2dfc15732474de"
server_key_iv: "e04ee008acb46fee78ab4aa97a7779fb"
server_key_version: "encryptionkey@0"
I20260620 11:33:18.614851 4489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:33:18.615667 4504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:18.615840 4489 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:33:18.615938 4489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "e6987b754865402bbf0d19c7bbe9c1e9"
format_stamp: "Formatted at 2026-06-20 11:33:18 on dist-test-slave-ctbw"
server_key: "24ba1cb8248796ed9b2dfc15732474de"
server_key_iv: "e04ee008acb46fee78ab4aa97a7779fb"
server_key_version: "encryptionkey@0"
I20260620 11:33:18.616019 4489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:18.639554 4489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:18.641605 4489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:18.641775 4489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:18.642315 4489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:18.642385 4489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:18.642421 4489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:18.642465 4489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:18.649156 4489 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:38635
I20260620 11:33:18.649240 4619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:38635 every 8 connection(s)
Jun 20 11:33:18 dist-test-slave-ctbw krb5kdc[4092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955198, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:18.649559 4489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:33:18.652747 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:18.642761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:47517 (local address 127.2.243.190:41559)
0620 11:33:18.642841 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:18.642844 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:18.643076 (+ 232us) server_negotiation.cc:407] Connection header received
0620 11:33:18.643551 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:18.643552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:18.643566 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:18.643652 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:18.644516 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.645356 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.646940 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.647267 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.647350 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:18.650007 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:18.650008 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:18.650008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:18.650015 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:18.651382 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:18.651777 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:18.651778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:18.651779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:18.651847 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:18.652158 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:18.652159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:18.652159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:18.652275 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:18.652369 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:18.652602 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 11:33:18.652684 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 11:33:18.652832 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:18.642754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41559 (local address 127.2.243.131:47517)
0620 11:33:18.642938 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:18.642976 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:18.643451 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:18.643736 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:18.643740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:18.644093 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:18.644396 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:18.644415 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.645469 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.645469 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:18.646823 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:18.646826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.647375 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.647376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:18.647481 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:18.648052 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:18.648063 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:18.649921 (+ 1858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:18.651459 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:18.651464 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:18.651466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:18.651681 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:18.651944 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:18.651945 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:18.651946 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:18.652070 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:18.652359 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:18.652362 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:18.652468 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 11:33:18.652550 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 11:33:18.652668 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
I20260620 11:33:18.653398 4620 heartbeater.cc:347] Connected to a master server at 127.2.243.190:41559
I20260620 11:33:18.653486 4620 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:18.653666 4620 heartbeater.cc:511] Master 127.2.243.190:41559 requested a full tablet report, sending...
I20260620 11:33:18.654029 4136 ts_manager.cc:194] Registered new tserver with Master: e6987b754865402bbf0d19c7bbe9c1e9 (127.2.243.131:38635)
I20260620 11:33:18.654510 4136 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:47517
I20260620 11:33:18.659155 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 4489
I20260620 11:33:18.659267 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:33:18.659484 3022 external_mini_cluster.cc:1428] Setting key 0e9036920eadbcc7b107d63f590e5ef4
I20260620 11:33:18.660571 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:33:18.660853 4626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:18.654409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41559 (local address 127.2.243.131:57873)
0620 11:33:18.654666 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:18.654678 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:18.654731 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:18.654916 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:18.654916 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:18.655095 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:18.655159 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:18.655161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.655893 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.655893 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:18.656365 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:18.656366 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.656732 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.656733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:18.656785 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:18.657192 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:18.657196 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:18.657482 (+ 286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:18.659637 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:18.659638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:18.659639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:18.659840 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:18.660104 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:18.660105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:18.660105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:18.660177 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:18.660496 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:18.660497 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:18.660559 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 11:33:18.660641 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 11:33:18.660764 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":75,"threads_started":1}
I20260620 11:33:18.660877 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:18.654532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:57873 (local address 127.2.243.190:41559)
0620 11:33:18.654577 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:18.654579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:18.654717 (+ 138us) server_negotiation.cc:407] Connection header received
0620 11:33:18.654784 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:18.654785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:18.654793 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:18.654835 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:18.655242 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.655790 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.656473 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.656664 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.656723 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:18.657583 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:18.657584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:18.657584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:18.657590 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:18.659543 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:18.659960 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:18.659961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:18.659961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:18.660032 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:18.660248 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:18.660249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:18.660249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:18.660401 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:18.660503 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:18.660720 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 11:33:18.660797 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 11:33:18 dist-test-slave-ctbw krb5kdc[4092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955198, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 11:33:18 dist-test-slave-ctbw krb5kdc[4092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955198, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:18.695290 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:18.687249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46968 (local address 127.2.243.190:41559)
0620 11:33:18.687341 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:18.687344 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:18.687357 (+ 13us) server_negotiation.cc:407] Connection header received
0620 11:33:18.687390 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:18.687391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:18.687405 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:18.687496 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:18.688421 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.689174 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.690065 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.690253 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.690327 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:18.692757 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:18.692758 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:18.692758 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:18.692765 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:18.694262 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:18.694604 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:18.694605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:18.694605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:18.694656 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:18.694831 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:18.694831 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:18.694831 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:18.694986 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:18.695058 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:18.695164 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 11:33:18.695222 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 11:33:18.697121 4136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:33:18.698678 4136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:33:18.705348 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:18.701228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:37443 (local address 127.0.0.1:47646)
0620 11:33:18.701967 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:18.701977 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:18.702027 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:18.702181 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:18.702181 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:18.702183 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:18.702224 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:18.702226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.703150 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.703151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:18.704357 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:18.704358 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.705038 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.705038 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:18.705113 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:18.705127 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 11:33:18.705197 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 11:33:18.705245 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":719,"mutex_wait_us":58,"thread_start_us":60,"threads_started":1}
I20260620 11:33:18.705348 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:18.701010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:38635 (local address 127.0.0.1:49200)
0620 11:33:18.701188 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:18.701231 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:18.701299 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:18.701876 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:18.701877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:18.701879 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:18.701916 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:18.701918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.703131 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.703132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:18.704150 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:18.704152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.705076 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.705077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:18.705142 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:18.705147 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:33:18.705179 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 11:33:18.705243 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":45,"thread_start_us":51,"threads_started":1}
I20260620 11:33:18.705564 4640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:18.701102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55072 (local address 127.2.243.129:40897)
0620 11:33:18.701444 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:18.701449 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:18.701460 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:33:18.701499 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:18.701501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:18.701517 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:18.701597 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:18.701777 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.702956 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.704436 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.705126 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.705194 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:18.705249 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:18.705401 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 11:33:18.705441 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":83,"threads_started":1}
I20260620 11:33:18.705687 4639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:18.701083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.2.243.131:38635)
0620 11:33:18.701718 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:18.701721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:18.701731 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:33:18.701766 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:18.701768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:18.701784 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:18.701878 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:18.701998 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.703065 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.704238 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.704969 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.705435 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:18.705499 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:18.705538 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 11:33:18.705581 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":72,"threads_started":1}
I20260620 11:33:18.705829 4637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:18.701008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:40897 (local address 127.0.0.1:55072)
0620 11:33:18.701274 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:18.701284 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:18.701333 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:18.701624 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:18.701626 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:18.701628 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:18.701696 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:18.701697 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.703036 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.703037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:18.704350 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:18.704353 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.705194 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.705195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:18.705259 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:18.705263 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:33:18.705313 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 11:33:18.705357 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":2,"thread_start_us":84,"threads_started":1}
I20260620 11:33:18.705992 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:18.701318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47646 (local address 127.2.243.130:37443)
0620 11:33:18.701535 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:18.701538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:18.702024 (+ 486us) server_negotiation.cc:407] Connection header received
0620 11:33:18.702091 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:18.702093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:18.702107 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:18.702181 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:18.702276 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.703060 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.704449 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:18.704949 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:18.705039 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:18.705100 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:18.705288 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 11:33:18.705332 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":136,"threads_started":1}
I20260620 11:33:18.706604 4554 tablet_service.cc:1467] Processing CreateTablet for tablet 086a0747023b42d385337a8ef4fef3ae (DEFAULT_TABLE table=test-table [id=09d6cfa190de433ab4aa507d41c9ca31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:18.706571 4414 tablet_service.cc:1467] Processing CreateTablet for tablet 086a0747023b42d385337a8ef4fef3ae (DEFAULT_TABLE table=test-table [id=09d6cfa190de433ab4aa507d41c9ca31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:18.706571 4274 tablet_service.cc:1467] Processing CreateTablet for tablet 086a0747023b42d385337a8ef4fef3ae (DEFAULT_TABLE table=test-table [id=09d6cfa190de433ab4aa507d41c9ca31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:18.706982 4414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086a0747023b42d385337a8ef4fef3ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:18.707031 4274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086a0747023b42d385337a8ef4fef3ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:18.707052 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 086a0747023b42d385337a8ef4fef3ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:18.709499 4643 tablet_bootstrap.cc:492] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9: Bootstrap starting.
I20260620 11:33:18.710319 4643 tablet_bootstrap.cc:654] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:18.710697 4643 log.cc:824] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:18.710768 4645 tablet_bootstrap.cc:492] T 086a0747023b42d385337a8ef4fef3ae P d51f2981b6c34325adc2d0191d45f729: Bootstrap starting.
I20260620 11:33:18.711019 4644 tablet_bootstrap.cc:492] T 086a0747023b42d385337a8ef4fef3ae P dd7cab7e392f4ab2bf49e6f936c1ace1: Bootstrap starting.
I20260620 11:33:18.711417 4645 tablet_bootstrap.cc:654] T 086a0747023b42d385337a8ef4fef3ae P d51f2981b6c34325adc2d0191d45f729: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:18.711640 4644 tablet_bootstrap.cc:654] T 086a0747023b42d385337a8ef4fef3ae P dd7cab7e392f4ab2bf49e6f936c1ace1: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:18.711760 4645 log.cc:824] T 086a0747023b42d385337a8ef4fef3ae P d51f2981b6c34325adc2d0191d45f729: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:18.711941 4644 log.cc:824] T 086a0747023b42d385337a8ef4fef3ae P dd7cab7e392f4ab2bf49e6f936c1ace1: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:18.712639 4645 tablet_bootstrap.cc:492] T 086a0747023b42d385337a8ef4fef3ae P d51f2981b6c34325adc2d0191d45f729: No bootstrap required, opened a new log
I20260620 11:33:18.712710 4645 ts_tablet_manager.cc:1397] T 086a0747023b42d385337a8ef4fef3ae P d51f2981b6c34325adc2d0191d45f729: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:33:18.712743 4644 tablet_bootstrap.cc:492] T 086a0747023b42d385337a8ef4fef3ae P dd7cab7e392f4ab2bf49e6f936c1ace1: No bootstrap required, opened a new log
I20260620 11:33:18.712795 4644 ts_tablet_manager.cc:1397] T 086a0747023b42d385337a8ef4fef3ae P dd7cab7e392f4ab2bf49e6f936c1ace1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:33:18.714702 4645 raft_consensus.cc:348] T 086a0747023b42d385337a8ef4fef3ae P d51f2981b6c34325adc2d0191d45f729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:18.714702 4644 raft_consensus.cc:348] T 086a0747023b42d385337a8ef4fef3ae P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:18.714900 4644 raft_consensus.cc:374] T 086a0747023b42d385337a8ef4fef3ae P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:18.714922 4645 raft_consensus.cc:374] T 086a0747023b42d385337a8ef4fef3ae P d51f2981b6c34325adc2d0191d45f729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:18.714989 4645 raft_consensus.cc:729] T 086a0747023b42d385337a8ef4fef3ae P d51f2981b6c34325adc2d0191d45f729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d51f2981b6c34325adc2d0191d45f729, State: Initialized, Role: FOLLOWER
I20260620 11:33:18.714989 4644 raft_consensus.cc:729] T 086a0747023b42d385337a8ef4fef3ae P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7cab7e392f4ab2bf49e6f936c1ace1, State: Initialized, Role: FOLLOWER
I20260620 11:33:18.715111 4645 consensus_queue.cc:260] T 086a0747023b42d385337a8ef4fef3ae P d51f2981b6c34325adc2d0191d45f729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:18.715138 4644 consensus_queue.cc:260] T 086a0747023b42d385337a8ef4fef3ae P dd7cab7e392f4ab2bf49e6f936c1ace1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:18.715265 4643 tablet_bootstrap.cc:492] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9: No bootstrap required, opened a new log
I20260620 11:33:18.715340 4643 ts_tablet_manager.cc:1397] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260620 11:33:18.715394 4645 ts_tablet_manager.cc:1428] T 086a0747023b42d385337a8ef4fef3ae P d51f2981b6c34325adc2d0191d45f729: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260620 11:33:18.715394 4644 ts_tablet_manager.cc:1428] T 086a0747023b42d385337a8ef4fef3ae P dd7cab7e392f4ab2bf49e6f936c1ace1: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 11:33:18.715516 4480 heartbeater.cc:503] Master 127.2.243.190:41559 was elected leader, sending a full tablet report...
I20260620 11:33:18.715792 4340 heartbeater.cc:503] Master 127.2.243.190:41559 was elected leader, sending a full tablet report...
I20260620 11:33:18.717217 4643 raft_consensus.cc:348] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:18.717343 4643 raft_consensus.cc:374] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:18.717367 4643 raft_consensus.cc:729] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6987b754865402bbf0d19c7bbe9c1e9, State: Initialized, Role: FOLLOWER
I20260620 11:33:18.717501 4643 consensus_queue.cc:260] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:18.717777 4643 ts_tablet_manager.cc:1428] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 11:33:18.717808 4620 heartbeater.cc:503] Master 127.2.243.190:41559 was elected leader, sending a full tablet report...
W20260620 11:33:18.733042 4481 tablet.cc:2369] T 086a0747023b42d385337a8ef4fef3ae P d51f2981b6c34325adc2d0191d45f729: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 11:33:18.834205 4341 tablet.cc:2369] T 086a0747023b42d385337a8ef4fef3ae P dd7cab7e392f4ab2bf49e6f936c1ace1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 11:33:18.900188 4621 tablet.cc:2369] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:33:19.028671 4651 raft_consensus.cc:482] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:33:19.028796 4651 raft_consensus.cc:504] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:19.029171 4651 leader_election.cc:288] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7cab7e392f4ab2bf49e6f936c1ace1 (127.2.243.129:40897), d51f2981b6c34325adc2d0191d45f729 (127.2.243.130:37443)
I20260620 11:33:19.033569 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:19.029661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:33863 (local address 127.2.243.130:37443)
0620 11:33:19.029724 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:19.029726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:19.029847 (+ 121us) server_negotiation.cc:407] Connection header received
0620 11:33:19.029884 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:19.029884 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:19.029892 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:19.029977 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:19.030119 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.030807 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.032563 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.033127 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.033211 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:19.033219 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:19.033453 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 11:33:19.033498 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 11:33:19.033572 4640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:19.029496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:47991 (local address 127.2.243.129:40897)
0620 11:33:19.029570 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:19.029572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:19.029587 (+ 15us) server_negotiation.cc:407] Connection header received
0620 11:33:19.029671 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:19.029672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:19.029684 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:19.029776 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:19.029945 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.030814 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.032563 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.033127 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.033210 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:19.033219 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:19.033480 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 11:33:19.033516 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 11:33:19.033573 4513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:19.029406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:40897 (local address 127.2.243.131:47991)
0620 11:33:19.029495 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:19.029509 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:19.029560 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:19.029782 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:19.029783 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:19.029785 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:19.029861 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:19.029864 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.030938 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.030939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:19.032439 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:19.032442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.033239 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.033240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:19.033349 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:19.033353 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:33:19.033468 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 11:33:19.033498 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":102}
I20260620 11:33:19.033581 4652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:19.029593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:37443 (local address 127.2.243.131:33863)
0620 11:33:19.029755 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:19.029766 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:19.029821 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:19.029993 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:19.029993 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:19.029995 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:19.030049 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:19.030050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.030932 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.030934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:19.032439 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:19.032442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.033239 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.033241 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:19.033348 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:19.033354 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 11:33:19.033424 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 11:33:19.033479 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":94,"thread_start_us":99,"threads_started":1}
I20260620 11:33:19.033983 4434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086a0747023b42d385337a8ef4fef3ae" candidate_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d51f2981b6c34325adc2d0191d45f729" is_pre_election: true
I20260620 11:33:19.034029 4294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086a0747023b42d385337a8ef4fef3ae" candidate_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" is_pre_election: true
I20260620 11:33:19.034155 4434 raft_consensus.cc:2454] T 086a0747023b42d385337a8ef4fef3ae P d51f2981b6c34325adc2d0191d45f729 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6987b754865402bbf0d19c7bbe9c1e9 in term 0.
I20260620 11:33:19.034228 4294 raft_consensus.cc:2454] T 086a0747023b42d385337a8ef4fef3ae P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6987b754865402bbf0d19c7bbe9c1e9 in term 0.
I20260620 11:33:19.034444 4507 leader_election.cc:302] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d51f2981b6c34325adc2d0191d45f729, e6987b754865402bbf0d19c7bbe9c1e9; no voters:
I20260620 11:33:19.034580 4651 raft_consensus.cc:2781] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:33:19.034669 4651 raft_consensus.cc:482] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:33:19.034703 4651 raft_consensus.cc:3037] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:19.035785 4651 raft_consensus.cc:504] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:19.035930 4651 leader_election.cc:288] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9 [CANDIDATE]: Term 1 election: Requested vote from peers dd7cab7e392f4ab2bf49e6f936c1ace1 (127.2.243.129:40897), d51f2981b6c34325adc2d0191d45f729 (127.2.243.130:37443)
I20260620 11:33:19.036141 4294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086a0747023b42d385337a8ef4fef3ae" candidate_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1"
I20260620 11:33:19.036196 4434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "086a0747023b42d385337a8ef4fef3ae" candidate_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d51f2981b6c34325adc2d0191d45f729"
I20260620 11:33:19.036226 4294 raft_consensus.cc:3037] T 086a0747023b42d385337a8ef4fef3ae P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:19.036293 4434 raft_consensus.cc:3037] T 086a0747023b42d385337a8ef4fef3ae P d51f2981b6c34325adc2d0191d45f729 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:19.036988 4294 raft_consensus.cc:2454] T 086a0747023b42d385337a8ef4fef3ae P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6987b754865402bbf0d19c7bbe9c1e9 in term 1.
I20260620 11:33:19.037104 4434 raft_consensus.cc:2454] T 086a0747023b42d385337a8ef4fef3ae P d51f2981b6c34325adc2d0191d45f729 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6987b754865402bbf0d19c7bbe9c1e9 in term 1.
I20260620 11:33:19.037177 4507 leader_election.cc:302] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7cab7e392f4ab2bf49e6f936c1ace1, e6987b754865402bbf0d19c7bbe9c1e9; no voters:
I20260620 11:33:19.037295 4651 raft_consensus.cc:2781] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:19.037499 4651 raft_consensus.cc:686] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9 [term 1 LEADER]: Becoming Leader. State: Replica: e6987b754865402bbf0d19c7bbe9c1e9, State: Running, Role: LEADER
I20260620 11:33:19.037616 4651 consensus_queue.cc:237] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:19.038437 4137 catalog_manager.cc:5495] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6987b754865402bbf0d19c7bbe9c1e9 (127.2.243.131). New cstate: current_term: 1 leader_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } health_report { overall_health: HEALTHY } } }
I20260620 11:33:19.060235 4639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:19.057784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49214 (local address 127.2.243.131:38635)
0620 11:33:19.057839 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:19.057841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:19.057878 (+ 37us) server_negotiation.cc:407] Connection header received
0620 11:33:19.057910 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:19.057911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:19.057931 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:19.057983 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:19.058119 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.058581 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.059475 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.059658 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.059720 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:19.059942 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:19.060032 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:19.060090 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:19.060143 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 11:33:19.060162 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 11:33:19.062042 4434 raft_consensus.cc:1261] T 086a0747023b42d385337a8ef4fef3ae P d51f2981b6c34325adc2d0191d45f729 [term 1 FOLLOWER]: Refusing update from remote peer e6987b754865402bbf0d19c7bbe9c1e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:19.062042 4294 raft_consensus.cc:1261] T 086a0747023b42d385337a8ef4fef3ae P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 1 FOLLOWER]: Refusing update from remote peer e6987b754865402bbf0d19c7bbe9c1e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:19.062382 4653 consensus_queue.cc:1035] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:19.062528 4651 consensus_queue.cc:1035] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:19.064916 4656 mvcc.cc:205] Tried to move back new op lower bound from 7298888495354662912 to 7298888495258329088. Current Snapshot: MvccSnapshot[applied={T|T < 7298888495354662912}]
I20260620 11:33:19.065668 4658 mvcc.cc:205] Tried to move back new op lower bound from 7298888495354662912 to 7298888495258329088. Current Snapshot: MvccSnapshot[applied={T|T < 7298888495354662912 or (T in {7298888495354662912})}]
I20260620 11:33:19.066064 4657 mvcc.cc:205] Tried to move back new op lower bound from 7298888495354662912 to 7298888495258329088. Current Snapshot: MvccSnapshot[applied={T|T < 7298888495354662912}]
I20260620 11:33:19.068621 4136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:19.068696 4136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:19.069559 4136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2: Sending DeleteTablet for 3 replicas of tablet 086a0747023b42d385337a8ef4fef3ae
I20260620 11:33:19.069908 4554 tablet_service.cc:1514] Processing DeleteTablet for tablet 086a0747023b42d385337a8ef4fef3ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:19 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:49200
I20260620 11:33:19.069936 4274 tablet_service.cc:1514] Processing DeleteTablet for tablet 086a0747023b42d385337a8ef4fef3ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:19 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:55072
I20260620 11:33:19.070152 4666 tablet_replica.cc:330] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9: stopping tablet replica
I20260620 11:33:19.070202 4414 tablet_service.cc:1514] Processing DeleteTablet for tablet 086a0747023b42d385337a8ef4fef3ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:19 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:47646
I20260620 11:33:19.070194 4136 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 11:33:19.070266 4666 raft_consensus.cc:2229] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9 [term 1 LEADER]: Raft consensus shutting down.
W20260620 11:33:19.070333 4136 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:33:19.070354 4666 raft_consensus.cc:2258] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:19.070843 4667 tablet_replica.cc:330] T 086a0747023b42d385337a8ef4fef3ae P dd7cab7e392f4ab2bf49e6f936c1ace1: stopping tablet replica
I20260620 11:33:19.071113 4667 raft_consensus.cc:2229] T 086a0747023b42d385337a8ef4fef3ae P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:19.071249 4666 ts_tablet_manager.cc:1905] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:19.071641 4554 tablet_service.cc:1467] Processing CreateTablet for tablet 9d104b5a1dcd4bbe90ce0a6e3a437042 (DEFAULT_TABLE table=test-table [id=04553efbc6fa40edbe6bb06283eac02a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:19.071704 4414 tablet_service.cc:1467] Processing CreateTablet for tablet 9d104b5a1dcd4bbe90ce0a6e3a437042 (DEFAULT_TABLE table=test-table [id=04553efbc6fa40edbe6bb06283eac02a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:19.071712 4274 tablet_service.cc:1467] Processing CreateTablet for tablet 9d104b5a1dcd4bbe90ce0a6e3a437042 (DEFAULT_TABLE table=test-table [id=04553efbc6fa40edbe6bb06283eac02a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:19.071755 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d104b5a1dcd4bbe90ce0a6e3a437042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:19.071800 4414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d104b5a1dcd4bbe90ce0a6e3a437042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:19.071823 4274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d104b5a1dcd4bbe90ce0a6e3a437042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:19.073096 4643 tablet_bootstrap.cc:492] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P e6987b754865402bbf0d19c7bbe9c1e9: Bootstrap starting.
I20260620 11:33:19.073403 4644 tablet_bootstrap.cc:492] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1: Bootstrap starting.
I20260620 11:33:19.073436 4645 tablet_bootstrap.cc:492] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P d51f2981b6c34325adc2d0191d45f729: Bootstrap starting.
I20260620 11:33:19.073521 4643 tablet_bootstrap.cc:654] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P e6987b754865402bbf0d19c7bbe9c1e9: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:19.073688 4666 ts_tablet_manager.cc:1918] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:33:19.073772 4644 tablet_bootstrap.cc:654] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:19.073845 4645 tablet_bootstrap.cc:654] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P d51f2981b6c34325adc2d0191d45f729: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:19.073861 4668 tablet_replica.cc:330] T 086a0747023b42d385337a8ef4fef3ae P d51f2981b6c34325adc2d0191d45f729: stopping tablet replica
I20260620 11:33:19.073974 4668 raft_consensus.cc:2229] T 086a0747023b42d385337a8ef4fef3ae P d51f2981b6c34325adc2d0191d45f729 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:19.074494 4668 raft_consensus.cc:2258] T 086a0747023b42d385337a8ef4fef3ae P d51f2981b6c34325adc2d0191d45f729 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:19.074944 4666 log.cc:1196] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/086a0747023b42d385337a8ef4fef3ae
I20260620 11:33:19.075209 4645 tablet_bootstrap.cc:492] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P d51f2981b6c34325adc2d0191d45f729: No bootstrap required, opened a new log
I20260620 11:33:19.075264 4666 ts_tablet_manager.cc:1939] T 086a0747023b42d385337a8ef4fef3ae P e6987b754865402bbf0d19c7bbe9c1e9: Deleting consensus metadata
I20260620 11:33:19.075273 4645 ts_tablet_manager.cc:1397] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P d51f2981b6c34325adc2d0191d45f729: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 11:33:19.075457 4645 raft_consensus.cc:348] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P d51f2981b6c34325adc2d0191d45f729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:19.075526 4645 raft_consensus.cc:374] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P d51f2981b6c34325adc2d0191d45f729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:19.075546 4645 raft_consensus.cc:729] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P d51f2981b6c34325adc2d0191d45f729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d51f2981b6c34325adc2d0191d45f729, State: Initialized, Role: FOLLOWER
I20260620 11:33:19.075596 4645 consensus_queue.cc:260] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P d51f2981b6c34325adc2d0191d45f729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:19.075681 4124 catalog_manager.cc:4841] TS e6987b754865402bbf0d19c7bbe9c1e9 (127.2.243.131:38635): tablet 086a0747023b42d385337a8ef4fef3ae (table test-table [id=09d6cfa190de433ab4aa507d41c9ca31]) successfully deleted
I20260620 11:33:19.075721 4645 ts_tablet_manager.cc:1428] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P d51f2981b6c34325adc2d0191d45f729: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:19.076005 4667 raft_consensus.cc:2258] T 086a0747023b42d385337a8ef4fef3ae P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:19.076023 4668 ts_tablet_manager.cc:1905] T 086a0747023b42d385337a8ef4fef3ae P d51f2981b6c34325adc2d0191d45f729: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:19.076535 4667 ts_tablet_manager.cc:1905] T 086a0747023b42d385337a8ef4fef3ae P dd7cab7e392f4ab2bf49e6f936c1ace1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:19.078080 4668 ts_tablet_manager.cc:1918] T 086a0747023b42d385337a8ef4fef3ae P d51f2981b6c34325adc2d0191d45f729: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:33:19.078163 4668 log.cc:1196] T 086a0747023b42d385337a8ef4fef3ae P d51f2981b6c34325adc2d0191d45f729: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-1/wal/wals/086a0747023b42d385337a8ef4fef3ae
I20260620 11:33:19.078454 4668 ts_tablet_manager.cc:1939] T 086a0747023b42d385337a8ef4fef3ae P d51f2981b6c34325adc2d0191d45f729: Deleting consensus metadata
I20260620 11:33:19.078572 4667 ts_tablet_manager.cc:1918] T 086a0747023b42d385337a8ef4fef3ae P dd7cab7e392f4ab2bf49e6f936c1ace1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:33:19.078682 4667 log.cc:1196] T 086a0747023b42d385337a8ef4fef3ae P dd7cab7e392f4ab2bf49e6f936c1ace1: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-0/wal/wals/086a0747023b42d385337a8ef4fef3ae
I20260620 11:33:19.078925 4123 catalog_manager.cc:4841] TS d51f2981b6c34325adc2d0191d45f729 (127.2.243.130:37443): tablet 086a0747023b42d385337a8ef4fef3ae (table test-table [id=09d6cfa190de433ab4aa507d41c9ca31]) successfully deleted
I20260620 11:33:19.079171 4644 tablet_bootstrap.cc:492] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1: No bootstrap required, opened a new log
I20260620 11:33:19.079223 4644 ts_tablet_manager.cc:1397] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260620 11:33:19.079245 4643 tablet_bootstrap.cc:492] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P e6987b754865402bbf0d19c7bbe9c1e9: No bootstrap required, opened a new log
I20260620 11:33:19.079283 4667 ts_tablet_manager.cc:1939] T 086a0747023b42d385337a8ef4fef3ae P dd7cab7e392f4ab2bf49e6f936c1ace1: Deleting consensus metadata
I20260620 11:33:19.079288 4643 ts_tablet_manager.cc:1397] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P e6987b754865402bbf0d19c7bbe9c1e9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260620 11:33:19.079435 4643 raft_consensus.cc:348] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P e6987b754865402bbf0d19c7bbe9c1e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:19.079499 4643 raft_consensus.cc:374] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P e6987b754865402bbf0d19c7bbe9c1e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:19.079519 4643 raft_consensus.cc:729] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P e6987b754865402bbf0d19c7bbe9c1e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6987b754865402bbf0d19c7bbe9c1e9, State: Initialized, Role: FOLLOWER
I20260620 11:33:19.079561 4643 consensus_queue.cc:260] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P e6987b754865402bbf0d19c7bbe9c1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:19.079635 4643 ts_tablet_manager.cc:1428] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P e6987b754865402bbf0d19c7bbe9c1e9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:19.079577 4644 raft_consensus.cc:348] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:19.079663 4644 raft_consensus.cc:374] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:19.079690 4644 raft_consensus.cc:729] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7cab7e392f4ab2bf49e6f936c1ace1, State: Initialized, Role: FOLLOWER
I20260620 11:33:19.079704 4123 catalog_manager.cc:4841] TS dd7cab7e392f4ab2bf49e6f936c1ace1 (127.2.243.129:40897): tablet 086a0747023b42d385337a8ef4fef3ae (table test-table [id=09d6cfa190de433ab4aa507d41c9ca31]) successfully deleted
I20260620 11:33:19.079735 4644 consensus_queue.cc:260] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:19.079840 4644 ts_tablet_manager.cc:1428] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:19.109606 4649 raft_consensus.cc:482] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:33:19.109719 4649 raft_consensus.cc:504] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:19.110076 4649 leader_election.cc:288] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d51f2981b6c34325adc2d0191d45f729 (127.2.243.130:37443), e6987b754865402bbf0d19c7bbe9c1e9 (127.2.243.131:38635)
I20260620 11:33:19.113801 4639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:19.110460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:48785 (local address 127.2.243.131:38635)
0620 11:33:19.110500 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:19.110502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:19.110510 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:33:19.110537 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:19.110537 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:19.110547 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:19.110606 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:19.110766 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.111524 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.113001 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.113449 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.113520 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:19.113528 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:19.113705 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 11:33:19.113736 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:33:19.113827 4672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:19.110255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:38635 (local address 127.2.243.129:48785)
0620 11:33:19.110387 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:19.110395 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:19.110447 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:19.110607 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:19.110608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:19.110610 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:19.110684 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:19.110686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.111605 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.111605 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:19.112877 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:19.112879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.113560 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.113561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:19.113621 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:19.113624 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:33:19.113686 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 11:33:19.113727 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":30,"thread_start_us":72,"threads_started":1}
I20260620 11:33:19.114081 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:19.110297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:54239 (local address 127.2.243.130:37443)
0620 11:33:19.110341 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:19.110343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:19.110445 (+ 102us) server_negotiation.cc:407] Connection header received
0620 11:33:19.110569 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:19.110570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:19.110580 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:19.110629 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:19.110803 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.111337 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.112973 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.113680 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.113751 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:19.113759 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:19.113973 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 11:33:19.114011 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:33:19.114223 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d104b5a1dcd4bbe90ce0a6e3a437042" candidate_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" is_pre_election: true
I20260620 11:33:19.114284 4673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:19.110221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:37443 (local address 127.2.243.129:54239)
0620 11:33:19.110368 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:19.110380 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:19.110468 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:19.110667 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:19.110668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:19.110669 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:19.110722 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:19.110723 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.111487 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.111488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:19.112877 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:19.112879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.113812 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.113814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:19.113886 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:19.113890 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:33:19.114161 (+ 271us) client_negotiation.cc:240] Negotiation successful
0620 11:33:19.114202 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":35,"thread_start_us":58,"threads_started":1}
I20260620 11:33:19.114324 4574 raft_consensus.cc:2454] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P e6987b754865402bbf0d19c7bbe9c1e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7cab7e392f4ab2bf49e6f936c1ace1 in term 0.
I20260620 11:33:19.114586 4228 leader_election.cc:302] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7cab7e392f4ab2bf49e6f936c1ace1, e6987b754865402bbf0d19c7bbe9c1e9; no voters:
I20260620 11:33:19.114598 4434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d104b5a1dcd4bbe90ce0a6e3a437042" candidate_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d51f2981b6c34325adc2d0191d45f729" is_pre_election: true
I20260620 11:33:19.114696 4434 raft_consensus.cc:2454] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P d51f2981b6c34325adc2d0191d45f729 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd7cab7e392f4ab2bf49e6f936c1ace1 in term 0.
I20260620 11:33:19.114732 4649 raft_consensus.cc:2781] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:33:19.114784 4649 raft_consensus.cc:482] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:33:19.114815 4649 raft_consensus.cc:3037] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:19.115548 4649 raft_consensus.cc:504] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:19.115706 4649 leader_election.cc:288] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1 [CANDIDATE]: Term 1 election: Requested vote from peers d51f2981b6c34325adc2d0191d45f729 (127.2.243.130:37443), e6987b754865402bbf0d19c7bbe9c1e9 (127.2.243.131:38635)
I20260620 11:33:19.115839 4574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d104b5a1dcd4bbe90ce0a6e3a437042" candidate_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6987b754865402bbf0d19c7bbe9c1e9"
I20260620 11:33:19.115909 4574 raft_consensus.cc:3037] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P e6987b754865402bbf0d19c7bbe9c1e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:19.116590 4574 raft_consensus.cc:2454] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P e6987b754865402bbf0d19c7bbe9c1e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7cab7e392f4ab2bf49e6f936c1ace1 in term 1.
I20260620 11:33:19.116714 4434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d104b5a1dcd4bbe90ce0a6e3a437042" candidate_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d51f2981b6c34325adc2d0191d45f729"
I20260620 11:33:19.116791 4434 raft_consensus.cc:3037] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P d51f2981b6c34325adc2d0191d45f729 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:19.116786 4228 leader_election.cc:302] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7cab7e392f4ab2bf49e6f936c1ace1, e6987b754865402bbf0d19c7bbe9c1e9; no voters:
I20260620 11:33:19.116881 4649 raft_consensus.cc:2781] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:19.117041 4649 raft_consensus.cc:686] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 1 LEADER]: Becoming Leader. State: Replica: dd7cab7e392f4ab2bf49e6f936c1ace1, State: Running, Role: LEADER
I20260620 11:33:19.117106 4649 consensus_queue.cc:237] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:19.117594 4434 raft_consensus.cc:2454] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P d51f2981b6c34325adc2d0191d45f729 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd7cab7e392f4ab2bf49e6f936c1ace1 in term 1.
I20260620 11:33:19.117960 4137 catalog_manager.cc:5495] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd7cab7e392f4ab2bf49e6f936c1ace1 (127.2.243.129). New cstate: current_term: 1 leader_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } health_report { overall_health: UNKNOWN } } }
Jun 20 11:33:19 dist-test-slave-ctbw krb5kdc[4092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955198, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:19.156433 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:19.147792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.2.243.190:41559)
0620 11:33:19.147849 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:19.147852 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:19.147972 (+ 120us) server_negotiation.cc:407] Connection header received
0620 11:33:19.148004 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:19.148005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:19.148017 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:19.148099 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:19.148621 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.149408 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.150142 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.150437 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.150518 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:19.153151 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:19.153153 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:19.153153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:19.153161 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:19.154855 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:19.155416 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:19.155417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:19.155418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:19.155490 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:19.155723 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:19.155724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:19.155724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:19.155860 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:19.155944 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:19.156150 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 11:33:19.156345 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":32}
I20260620 11:33:19.156435 4681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:19.147711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41559 (local address 127.0.0.1:46980)
0620 11:33:19.147870 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:19.147882 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:19.147959 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:19.148111 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:19.148111 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:19.148462 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:19.148513 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:19.148516 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.149507 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.149508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:19.150042 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:19.150044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.150538 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.150539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:19.150636 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:19.151265 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:19.151270 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:19.153053 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:19.154994 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:19.155004 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:19.155006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:19.155249 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:19.155574 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:19.155575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:19.155576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:19.155643 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:19.155933 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:19.155942 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:19.155998 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 11:33:19.156119 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 11:33:19.156284 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260620 11:33:19.160281 4137 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:46980:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 11:33:19.160502 4137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:33:19.162014 4554 tablet_service.cc:1467] Processing CreateTablet for tablet ba9e4237b0a44c45a7c169263b5b6c53 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=eaa11967c0e64b69bf5bf3ca558f57e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 11:33:19.162009 4414 tablet_service.cc:1467] Processing CreateTablet for tablet ba9e4237b0a44c45a7c169263b5b6c53 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=eaa11967c0e64b69bf5bf3ca558f57e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 11:33:19.162029 4274 tablet_service.cc:1467] Processing CreateTablet for tablet ba9e4237b0a44c45a7c169263b5b6c53 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=eaa11967c0e64b69bf5bf3ca558f57e1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 11:33:19.162168 4554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba9e4237b0a44c45a7c169263b5b6c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:19.162168 4414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba9e4237b0a44c45a7c169263b5b6c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:19.162184 4274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba9e4237b0a44c45a7c169263b5b6c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:19.163666 4645 tablet_bootstrap.cc:492] T ba9e4237b0a44c45a7c169263b5b6c53 P d51f2981b6c34325adc2d0191d45f729: Bootstrap starting.
I20260620 11:33:19.163727 4643 tablet_bootstrap.cc:492] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9: Bootstrap starting.
I20260620 11:33:19.163739 4644 tablet_bootstrap.cc:492] T ba9e4237b0a44c45a7c169263b5b6c53 P dd7cab7e392f4ab2bf49e6f936c1ace1: Bootstrap starting.
I20260620 11:33:19.164088 4645 tablet_bootstrap.cc:654] T ba9e4237b0a44c45a7c169263b5b6c53 P d51f2981b6c34325adc2d0191d45f729: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:19.164088 4643 tablet_bootstrap.cc:654] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:19.164145 4644 tablet_bootstrap.cc:654] T ba9e4237b0a44c45a7c169263b5b6c53 P dd7cab7e392f4ab2bf49e6f936c1ace1: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:19.164948 4644 tablet_bootstrap.cc:492] T ba9e4237b0a44c45a7c169263b5b6c53 P dd7cab7e392f4ab2bf49e6f936c1ace1: No bootstrap required, opened a new log
I20260620 11:33:19.164978 4645 tablet_bootstrap.cc:492] T ba9e4237b0a44c45a7c169263b5b6c53 P d51f2981b6c34325adc2d0191d45f729: No bootstrap required, opened a new log
I20260620 11:33:19.165000 4644 ts_tablet_manager.cc:1397] T ba9e4237b0a44c45a7c169263b5b6c53 P dd7cab7e392f4ab2bf49e6f936c1ace1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:19.165021 4645 ts_tablet_manager.cc:1397] T ba9e4237b0a44c45a7c169263b5b6c53 P d51f2981b6c34325adc2d0191d45f729: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 11:33:19.165143 4644 raft_consensus.cc:348] T ba9e4237b0a44c45a7c169263b5b6c53 P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:19.165166 4645 raft_consensus.cc:348] T ba9e4237b0a44c45a7c169263b5b6c53 P d51f2981b6c34325adc2d0191d45f729 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:19.165223 4644 raft_consensus.cc:374] T ba9e4237b0a44c45a7c169263b5b6c53 P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:19.165243 4644 raft_consensus.cc:729] T ba9e4237b0a44c45a7c169263b5b6c53 P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd7cab7e392f4ab2bf49e6f936c1ace1, State: Initialized, Role: FOLLOWER
I20260620 11:33:19.165243 4645 raft_consensus.cc:374] T ba9e4237b0a44c45a7c169263b5b6c53 P d51f2981b6c34325adc2d0191d45f729 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:19.165269 4645 raft_consensus.cc:729] T ba9e4237b0a44c45a7c169263b5b6c53 P d51f2981b6c34325adc2d0191d45f729 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d51f2981b6c34325adc2d0191d45f729, State: Initialized, Role: FOLLOWER
I20260620 11:33:19.165285 4644 consensus_queue.cc:260] T ba9e4237b0a44c45a7c169263b5b6c53 P dd7cab7e392f4ab2bf49e6f936c1ace1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:19.165314 4645 consensus_queue.cc:260] T ba9e4237b0a44c45a7c169263b5b6c53 P d51f2981b6c34325adc2d0191d45f729 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:19.165372 4644 ts_tablet_manager.cc:1428] T ba9e4237b0a44c45a7c169263b5b6c53 P dd7cab7e392f4ab2bf49e6f936c1ace1: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 11:33:19.165426 4645 ts_tablet_manager.cc:1428] T ba9e4237b0a44c45a7c169263b5b6c53 P d51f2981b6c34325adc2d0191d45f729: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:19.165526 4650 tablet_replica.cc:439] T ba9e4237b0a44c45a7c169263b5b6c53 P d51f2981b6c34325adc2d0191d45f729: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } } }
I20260620 11:33:19.165603 4650 tablet_replica.cc:442] T ba9e4237b0a44c45a7c169263b5b6c53 P d51f2981b6c34325adc2d0191d45f729: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:33:19.165724 4643 tablet_bootstrap.cc:492] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9: No bootstrap required, opened a new log
I20260620 11:33:19.165767 4643 ts_tablet_manager.cc:1397] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 11:33:19.165940 4643 raft_consensus.cc:348] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:19.166010 4643 raft_consensus.cc:374] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:19.166034 4643 raft_consensus.cc:729] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6987b754865402bbf0d19c7bbe9c1e9, State: Initialized, Role: FOLLOWER
I20260620 11:33:19.166081 4643 consensus_queue.cc:260] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:19.166173 4643 ts_tablet_manager.cc:1428] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:19.166200 4651 tablet_replica.cc:439] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } } }
I20260620 11:33:19.166301 4651 tablet_replica.cc:442] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:33:19.166599 4649 tablet_replica.cc:439] T ba9e4237b0a44c45a7c169263b5b6c53 P dd7cab7e392f4ab2bf49e6f936c1ace1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } } }
I20260620 11:33:19.166684 4649 tablet_replica.cc:442] T ba9e4237b0a44c45a7c169263b5b6c53 P dd7cab7e392f4ab2bf49e6f936c1ace1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:33:19.303561 4651 raft_consensus.cc:482] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:33:19.303654 4651 raft_consensus.cc:504] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:19.303836 4651 leader_election.cc:288] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd7cab7e392f4ab2bf49e6f936c1ace1 (127.2.243.129:40897), d51f2981b6c34325adc2d0191d45f729 (127.2.243.130:37443)
I20260620 11:33:19.304095 4294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba9e4237b0a44c45a7c169263b5b6c53" candidate_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" is_pre_election: true
I20260620 11:33:19.304121 4434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba9e4237b0a44c45a7c169263b5b6c53" candidate_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d51f2981b6c34325adc2d0191d45f729" is_pre_election: true
I20260620 11:33:19.304207 4294 raft_consensus.cc:2454] T ba9e4237b0a44c45a7c169263b5b6c53 P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6987b754865402bbf0d19c7bbe9c1e9 in term 0.
I20260620 11:33:19.304235 4434 raft_consensus.cc:2454] T ba9e4237b0a44c45a7c169263b5b6c53 P d51f2981b6c34325adc2d0191d45f729 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6987b754865402bbf0d19c7bbe9c1e9 in term 0.
I20260620 11:33:19.304395 4507 leader_election.cc:302] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7cab7e392f4ab2bf49e6f936c1ace1, e6987b754865402bbf0d19c7bbe9c1e9; no voters:
I20260620 11:33:19.304522 4651 raft_consensus.cc:2781] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:33:19.304577 4651 raft_consensus.cc:482] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:33:19.304610 4651 raft_consensus.cc:3037] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:19.305296 4651 raft_consensus.cc:504] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:19.305435 4651 leader_election.cc:288] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9 [CANDIDATE]: Term 1 election: Requested vote from peers dd7cab7e392f4ab2bf49e6f936c1ace1 (127.2.243.129:40897), d51f2981b6c34325adc2d0191d45f729 (127.2.243.130:37443)
I20260620 11:33:19.305612 4434 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba9e4237b0a44c45a7c169263b5b6c53" candidate_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d51f2981b6c34325adc2d0191d45f729"
I20260620 11:33:19.305609 4294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba9e4237b0a44c45a7c169263b5b6c53" candidate_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1"
I20260620 11:33:19.305701 4434 raft_consensus.cc:3037] T ba9e4237b0a44c45a7c169263b5b6c53 P d51f2981b6c34325adc2d0191d45f729 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:19.305712 4294 raft_consensus.cc:3037] T ba9e4237b0a44c45a7c169263b5b6c53 P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:19.306283 4294 raft_consensus.cc:2454] T ba9e4237b0a44c45a7c169263b5b6c53 P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6987b754865402bbf0d19c7bbe9c1e9 in term 1.
I20260620 11:33:19.306401 4434 raft_consensus.cc:2454] T ba9e4237b0a44c45a7c169263b5b6c53 P d51f2981b6c34325adc2d0191d45f729 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6987b754865402bbf0d19c7bbe9c1e9 in term 1.
I20260620 11:33:19.306504 4507 leader_election.cc:302] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd7cab7e392f4ab2bf49e6f936c1ace1, e6987b754865402bbf0d19c7bbe9c1e9; no voters:
I20260620 11:33:19.306663 4651 raft_consensus.cc:2781] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:19.306743 4651 raft_consensus.cc:686] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9 [term 1 LEADER]: Becoming Leader. State: Replica: e6987b754865402bbf0d19c7bbe9c1e9, State: Running, Role: LEADER
I20260620 11:33:19.306802 4651 consensus_queue.cc:237] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } }
I20260620 11:33:19.307108 4653 tablet_replica.cc:439] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9: TxnStatusTablet state changed. Reason: New leader e6987b754865402bbf0d19c7bbe9c1e9. Latest consensus state: current_term: 1 leader_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } } }
I20260620 11:33:19.307183 4653 tablet_replica.cc:442] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9: This TxnStatusTablet replica's current role is: LEADER
I20260620 11:33:19.307384 4686 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 11:33:19.307413 4136 catalog_manager.cc:5495] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6987b754865402bbf0d19c7bbe9c1e9 (127.2.243.131). New cstate: current_term: 1 leader_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } health_report { overall_health: HEALTHY } } }
I20260620 11:33:19.307566 4686 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 11:33:19.318499 4681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:19.314950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:38635 (local address 127.0.0.1:49228)
0620 11:33:19.315036 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:19.315047 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:19.315080 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:19.315456 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:19.315457 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:19.315458 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:33:19.315512 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:19.315514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.316496 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.316497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:19.317434 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:19.317438 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.317971 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.317972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:19.318062 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:19.318082 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:33:19.318279 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:33:19.318287 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 11:33:19.318361 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 11:33:19.318411 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 11:33:19.318516 4639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:19.315248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49228 (local address 127.2.243.131:38635)
0620 11:33:19.315313 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:19.315315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:19.315327 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:33:19.315352 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:19.315353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:19.315364 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:19.315430 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:19.315599 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.316396 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.317530 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.317871 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.317966 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:19.318170 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:19.318209 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:19.318269 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:19.318381 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 11:33:19.318409 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:33:19.319788 4294 raft_consensus.cc:1261] T ba9e4237b0a44c45a7c169263b5b6c53 P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 1 FOLLOWER]: Refusing update from remote peer e6987b754865402bbf0d19c7bbe9c1e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:19.319840 4434 raft_consensus.cc:1261] T ba9e4237b0a44c45a7c169263b5b6c53 P d51f2981b6c34325adc2d0191d45f729 [term 1 FOLLOWER]: Refusing update from remote peer e6987b754865402bbf0d19c7bbe9c1e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:19.320235 4651 consensus_queue.cc:1035] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:19.320366 4653 consensus_queue.cc:1035] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:19.321017 4650 tablet_replica.cc:439] T ba9e4237b0a44c45a7c169263b5b6c53 P d51f2981b6c34325adc2d0191d45f729: TxnStatusTablet state changed. Reason: New leader e6987b754865402bbf0d19c7bbe9c1e9. Latest consensus state: current_term: 1 leader_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } } }
I20260620 11:33:19.321111 4650 tablet_replica.cc:442] T ba9e4237b0a44c45a7c169263b5b6c53 P d51f2981b6c34325adc2d0191d45f729: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:33:19.321295 4649 tablet_replica.cc:439] T ba9e4237b0a44c45a7c169263b5b6c53 P dd7cab7e392f4ab2bf49e6f936c1ace1: TxnStatusTablet state changed. Reason: New leader e6987b754865402bbf0d19c7bbe9c1e9. Latest consensus state: current_term: 1 leader_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } } }
I20260620 11:33:19.321382 4649 tablet_replica.cc:442] T ba9e4237b0a44c45a7c169263b5b6c53 P dd7cab7e392f4ab2bf49e6f936c1ace1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:33:19.321884 4653 tablet_replica.cc:439] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } } }
I20260620 11:33:19.321974 4653 tablet_replica.cc:442] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9: This TxnStatusTablet replica's current role is: LEADER
I20260620 11:33:19.322204 4650 tablet_replica.cc:439] T ba9e4237b0a44c45a7c169263b5b6c53 P d51f2981b6c34325adc2d0191d45f729: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } } }
I20260620 11:33:19.322288 4650 tablet_replica.cc:442] T ba9e4237b0a44c45a7c169263b5b6c53 P d51f2981b6c34325adc2d0191d45f729: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:33:19.327467 4653 tablet_replica.cc:439] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } } }
I20260620 11:33:19.327561 4653 tablet_replica.cc:442] T ba9e4237b0a44c45a7c169263b5b6c53 P e6987b754865402bbf0d19c7bbe9c1e9: This TxnStatusTablet replica's current role is: LEADER
I20260620 11:33:19.327932 4649 tablet_replica.cc:439] T ba9e4237b0a44c45a7c169263b5b6c53 P dd7cab7e392f4ab2bf49e6f936c1ace1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd7cab7e392f4ab2bf49e6f936c1ace1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 40897 } } peers { permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 } } peers { permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 } } }
I20260620 11:33:19.328025 4640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:19.324110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55086 (local address 127.2.243.129:40897)
0620 11:33:19.324151 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:19.324153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:19.324166 (+ 13us) server_negotiation.cc:407] Connection header received
0620 11:33:19.324281 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:19.324282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:19.324303 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:19.324370 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:19.324548 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.325396 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.326619 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.327002 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.327078 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:19.327299 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:19.327449 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:19.327523 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:19.327840 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 11:33:19.327865 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:33:19.328038 4649 tablet_replica.cc:442] T ba9e4237b0a44c45a7c169263b5b6c53 P dd7cab7e392f4ab2bf49e6f936c1ace1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:33:19.333942 4639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:19.330600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:57753 (local address 127.2.243.131:38635)
0620 11:33:19.330641 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:19.330643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:19.330652 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:33:19.330721 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:19.330722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:19.330736 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:19.330802 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:19.330976 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.331774 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.332994 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.333332 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.333408 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:19.333656 (+ 248us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:19.333695 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:19.333727 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:19.333842 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 11:33:19.333864 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:33:19.334316 4693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:19.330411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:38635 (local address 127.2.243.129:57753)
0620 11:33:19.330582 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:19.330594 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:19.330647 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:19.330806 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:19.330807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:19.330808 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:33:19.330911 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:19.330912 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.331866 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.331867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:19.332893 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:19.332896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.333451 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.333452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:19.333535 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:19.333551 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:33:19.333772 (+ 221us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:33:19.333773 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 11:33:19.334162 (+ 389us) client_negotiation.cc:240] Negotiation successful
0620 11:33:19.334215 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260620 11:33:19.336833 4434 raft_consensus.cc:1261] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P d51f2981b6c34325adc2d0191d45f729 [term 1 FOLLOWER]: Refusing update from remote peer dd7cab7e392f4ab2bf49e6f936c1ace1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:19.336916 4574 raft_consensus.cc:1261] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P e6987b754865402bbf0d19c7bbe9c1e9 [term 1 FOLLOWER]: Refusing update from remote peer dd7cab7e392f4ab2bf49e6f936c1ace1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:19.337173 4674 consensus_queue.cc:1035] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d51f2981b6c34325adc2d0191d45f729" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:19.337313 4649 consensus_queue.cc:1035] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6987b754865402bbf0d19c7bbe9c1e9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:19.349500 4698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:19.343378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:40897 (local address 127.2.243.131:46279)
0620 11:33:19.347005 (+ 3627us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:19.347018 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:19.347084 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:19.347197 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:19.347198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:19.347201 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:33:19.347254 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:19.347255 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.348009 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.348011 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:19.348669 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:19.348671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.349056 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.349057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:19.349113 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:19.349129 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:33:19.349323 (+ 194us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:33:19.349325 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:33:19.349375 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 11:33:19.349418 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3586,"thread_start_us":115,"threads_started":1}
I20260620 11:33:19.349479 4640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:19.343456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:46279 (local address 127.2.243.129:40897)
0620 11:33:19.343505 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:19.343506 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:19.347093 (+ 3587us) server_negotiation.cc:407] Connection header received
0620 11:33:19.347135 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:19.347136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:19.347147 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:19.347194 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:19.347308 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.347931 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.348759 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.348965 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.349055 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:19.349228 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:19.349262 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:19.349312 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:19.349380 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 11:33:19.349400 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:33:19.359746 4136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:19.359866 4136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:19.360533 4136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c3d7bf740d24cf2bedac5c416dd80b2: Sending DeleteTablet for 3 replicas of tablet 9d104b5a1dcd4bbe90ce0a6e3a437042
I20260620 11:33:19.360792 4274 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d104b5a1dcd4bbe90ce0a6e3a437042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:19 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:55072
I20260620 11:33:19.360884 4667 tablet_replica.cc:330] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1: stopping tablet replica
I20260620 11:33:19.360877 4414 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d104b5a1dcd4bbe90ce0a6e3a437042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:19 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:47646
I20260620 11:33:19.360939 4667 raft_consensus.cc:2229] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:33:19.360997 4668 tablet_replica.cc:330] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P d51f2981b6c34325adc2d0191d45f729: stopping tablet replica
I20260620 11:33:19.361022 4667 raft_consensus.cc:2258] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:19.361052 4668 raft_consensus.cc:2229] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P d51f2981b6c34325adc2d0191d45f729 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:19.361045 4554 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d104b5a1dcd4bbe90ce0a6e3a437042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:19 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:49200
I20260620 11:33:19.361092 4668 raft_consensus.cc:2258] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P d51f2981b6c34325adc2d0191d45f729 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:19.361228 4666 tablet_replica.cc:330] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P e6987b754865402bbf0d19c7bbe9c1e9: stopping tablet replica
I20260620 11:33:19.361285 4666 raft_consensus.cc:2229] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P e6987b754865402bbf0d19c7bbe9c1e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:19.361325 4666 raft_consensus.cc:2258] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P e6987b754865402bbf0d19c7bbe9c1e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:19.361595 4668 ts_tablet_manager.cc:1905] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P d51f2981b6c34325adc2d0191d45f729: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:19.361614 4667 ts_tablet_manager.cc:1905] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:19.361649 4666 ts_tablet_manager.cc:1905] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P e6987b754865402bbf0d19c7bbe9c1e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:19.363337 4666 ts_tablet_manager.cc:1918] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P e6987b754865402bbf0d19c7bbe9c1e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:33:19.363404 4666 log.cc:1196] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P e6987b754865402bbf0d19c7bbe9c1e9: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/9d104b5a1dcd4bbe90ce0a6e3a437042
I20260620 11:33:19.363636 4666 ts_tablet_manager.cc:1939] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P e6987b754865402bbf0d19c7bbe9c1e9: Deleting consensus metadata
I20260620 11:33:19.363932 4124 catalog_manager.cc:4841] TS e6987b754865402bbf0d19c7bbe9c1e9 (127.2.243.131:38635): tablet 9d104b5a1dcd4bbe90ce0a6e3a437042 (table test-table [id=04553efbc6fa40edbe6bb06283eac02a]) successfully deleted
I20260620 11:33:19.364223 4667 ts_tablet_manager.cc:1918] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:33:19.364269 4667 log.cc:1196] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-0/wal/wals/9d104b5a1dcd4bbe90ce0a6e3a437042
I20260620 11:33:19.364475 4667 ts_tablet_manager.cc:1939] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P dd7cab7e392f4ab2bf49e6f936c1ace1: Deleting consensus metadata
I20260620 11:33:19.364490 4668 ts_tablet_manager.cc:1918] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P d51f2981b6c34325adc2d0191d45f729: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:33:19.364557 4668 log.cc:1196] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P d51f2981b6c34325adc2d0191d45f729: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955192827595-3022-0/minicluster-data/ts-1/wal/wals/9d104b5a1dcd4bbe90ce0a6e3a437042
I20260620 11:33:19.364773 4668 ts_tablet_manager.cc:1939] T 9d104b5a1dcd4bbe90ce0a6e3a437042 P d51f2981b6c34325adc2d0191d45f729: Deleting consensus metadata
I20260620 11:33:19.365020 4123 catalog_manager.cc:4841] TS dd7cab7e392f4ab2bf49e6f936c1ace1 (127.2.243.129:40897): tablet 9d104b5a1dcd4bbe90ce0a6e3a437042 (table test-table [id=04553efbc6fa40edbe6bb06283eac02a]) successfully deleted
I20260620 11:33:19.365095 4123 catalog_manager.cc:4841] TS d51f2981b6c34325adc2d0191d45f729 (127.2.243.130:37443): tablet 9d104b5a1dcd4bbe90ce0a6e3a437042 (table test-table [id=04553efbc6fa40edbe6bb06283eac02a]) successfully deleted
Jun 20 11:33:19 dist-test-slave-ctbw krb5kdc[4092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955198, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.243.129@KRBTEST.COM
I20260620 11:33:19.382426 4640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:19.367160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55090 (local address 127.2.243.129:40897)
0620 11:33:19.367204 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:19.367207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:19.367353 (+ 146us) server_negotiation.cc:407] Connection header received
0620 11:33:19.367457 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:19.367458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:19.367469 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:19.367531 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:19.368222 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.368989 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.370038 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.370331 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.370405 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:19.379032 (+ 8627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:19.379042 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:19.379042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:19.379052 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:19.381085 (+ 2033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:19.381514 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:19.381518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:19.381518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:19.381599 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:19.381829 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:19.381830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:19.381830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:19.382011 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:19.382161 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:19.382235 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 11:33:19.382319 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 11:33:19.382778 4334 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55090
I20260620 11:33:19.391719 4186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:19.383825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46984 (local address 127.2.243.190:41559)
0620 11:33:19.383870 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:19.383873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:19.384103 (+ 230us) server_negotiation.cc:407] Connection header received
0620 11:33:19.384191 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:19.384192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:19.384204 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:19.384257 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:19.384875 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.385685 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.386772 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:19.387196 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:19.387272 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:19.388739 (+ 1467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:19.388741 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:19.388741 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:19.388749 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:19.390457 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:19.390856 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:19.390857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:19.390857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:19.390964 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:19.391182 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:19.391183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:19.391183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:19.391320 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:19.391427 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:19.391549 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 11:33:19.391627 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 11:33:19.392342 4136 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46984
I20260620 11:33:19.393095 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 4209
I20260620 11:33:19.398852 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 4349
I20260620 11:33:19.403657 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 4489
I20260620 11:33:19.409916 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 4105
2026-06-20T11:33:19Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2556 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:33:19 dist-test-slave-ctbw krb5kdc[4707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:33:19 dist-test-slave-ctbw krb5kdc[4707](info): set up 2 sockets
Jun 20 11:33:19 dist-test-slave-ctbw krb5kdc[4707](info): commencing operation
krb5kdc: starting...
W20260620 11:33:20.446995 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:33:20 dist-test-slave-ctbw krb5kdc[4707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:33:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:33:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:33:20.565727 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:34259
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:34259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:20.646466 4720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:20.646733 4720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:20.646770 4720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:20.648023 4720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:33:20.648105 4720 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 11:33:20.648132 4720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:20.648166 4720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:20.648203 4720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:33:20.648231 4720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:33:20.649708 4720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:34259
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:34259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:20.649997 4720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:20.650233 4720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:20.655184 4728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:20.655234 4725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:20.655194 4726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:20.655448 4720 server_base.cc:1034] running on GCE node
I20260620 11:33:20.655706 4720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:20.656018 4720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:20.657212 4720 hybrid_clock.cc:648] HybridClock initialized: now 1781955200657175 us; error 53 us; skew 500 ppm
Jun 20 11:33:20 dist-test-slave-ctbw krb5kdc[4707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955200, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:20.660058 4720 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:33:20.660485 4720 webserver.cc:460] Webserver started at http://127.2.243.190:32799/ using document root <none> and password file <none>
I20260620 11:33:20.660677 4720 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:20.660719 4720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:20.660813 4720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:20.661859 4720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "40ffddc373b64bfba32915ea932127b2"
format_stamp: "Formatted at 2026-06-20 11:33:20 on dist-test-slave-ctbw"
server_key: "c0a278f9f7b0394956a2fa4f7e4027b5"
server_key_iv: "5ff33fd5a7bfd7e164fb241ded86cd19"
server_key_version: "encryptionkey@0"
I20260620 11:33:20.662182 4720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "40ffddc373b64bfba32915ea932127b2"
format_stamp: "Formatted at 2026-06-20 11:33:20 on dist-test-slave-ctbw"
server_key: "c0a278f9f7b0394956a2fa4f7e4027b5"
server_key_iv: "5ff33fd5a7bfd7e164fb241ded86cd19"
server_key_version: "encryptionkey@0"
I20260620 11:33:20.663604 4720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:33:20.664381 4735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:20.664597 4720 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:20.664690 4720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "40ffddc373b64bfba32915ea932127b2"
format_stamp: "Formatted at 2026-06-20 11:33:20 on dist-test-slave-ctbw"
server_key: "c0a278f9f7b0394956a2fa4f7e4027b5"
server_key_iv: "5ff33fd5a7bfd7e164fb241ded86cd19"
server_key_version: "encryptionkey@0"
I20260620 11:33:20.664811 4720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:20.691201 4720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:20.693390 4720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:20.693552 4720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:20.698279 4720 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:34259
I20260620 11:33:20.698354 4797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:34259 every 8 connection(s)
I20260620 11:33:20.698707 4720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:33:20.699436 4798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:20.700671 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 4720
I20260620 11:33:20.700773 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:33:20.700942 3022 external_mini_cluster.cc:1428] Setting key ea8852d3dd9a13637c88d065546a0d9f
I20260620 11:33:20.702167 4798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2: Bootstrap starting.
I20260620 11:33:20.702754 4798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:20.703083 4798 log.cc:824] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:20.703783 4798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2: No bootstrap required, opened a new log
I20260620 11:33:20.705559 4798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ffddc373b64bfba32915ea932127b2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 34259 } }
I20260620 11:33:20.705780 4798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:20.705821 4798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40ffddc373b64bfba32915ea932127b2, State: Initialized, Role: FOLLOWER
I20260620 11:33:20.705963 4798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ffddc373b64bfba32915ea932127b2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 34259 } }
I20260620 11:33:20.706054 4798 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:33:20.706113 4798 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:33:20.706171 4798 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:20.706686 4798 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ffddc373b64bfba32915ea932127b2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 34259 } }
I20260620 11:33:20.706769 4798 leader_election.cc:302] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40ffddc373b64bfba32915ea932127b2; no voters:
I20260620 11:33:20.707033 4798 leader_election.cc:288] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:33:20.707206 4804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:20.707304 4798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:20.707362 4804 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2 [term 1 LEADER]: Becoming Leader. State: Replica: 40ffddc373b64bfba32915ea932127b2, State: Running, Role: LEADER
I20260620 11:33:20.707456 4804 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ffddc373b64bfba32915ea932127b2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 34259 } }
Jun 20 11:33:20 dist-test-slave-ctbw krb5kdc[4707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:20.707801 4804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40ffddc373b64bfba32915ea932127b2. Latest consensus state: current_term: 1 leader_uuid: "40ffddc373b64bfba32915ea932127b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ffddc373b64bfba32915ea932127b2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 34259 } } }
I20260620 11:33:20.707892 4804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2 [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:20.707815 4805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40ffddc373b64bfba32915ea932127b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40ffddc373b64bfba32915ea932127b2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 34259 } } }
I20260620 11:33:20.708042 4805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2 [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:20.708177 4811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:33:20.708686 4811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:33:20.710961 4811 catalog_manager.cc:1347] Generated new cluster ID: 7f9c2afaa87b46feacbabc7bdb54c4b9
I20260620 11:33:20.711014 4811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:33:20.712148 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:20.701478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37280 (local address 127.2.243.190:34259)
0620 11:33:20.701709 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:20.701716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:20.701741 (+ 25us) server_negotiation.cc:407] Connection header received
0620 11:33:20.702144 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:20.702150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:20.702197 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:20.702388 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:20.703057 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:20.704088 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:20.704859 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:20.705312 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:20.705435 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:20.708344 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:20.708348 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:20.708349 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:20.708363 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:20.710459 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:20.710830 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:20.710833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:20.710834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:20.710978 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:20.711211 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:20.711212 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:20.711212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:20.711522 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:20.711681 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:20.711806 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 11:33:20.711946 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
I20260620 11:33:20.721115 4811 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:33:20.721721 4811 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:33:20.726651 4811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2: Generated new TSK 0
I20260620 11:33:20.726845 4811 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
I20260620 11:33:20.757630 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:34259
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:20.850474 4824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:20.850683 4824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:20.850708 4824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:20.852020 4824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:20.852065 4824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:20.852085 4824 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 11:33:20.852097 4824 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 11:33:20.852120 4824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:33:20.853695 4824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.243.190:34259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:20.853957 4824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:20.854151 4824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:20.859102 4832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:20.859133 4829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:20.859180 4824 server_base.cc:1034] running on GCE node
W20260620 11:33:20.859102 4830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:20.859483 4824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:20.859754 4824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:20.860916 4824 hybrid_clock.cc:648] HybridClock initialized: now 1781955200860890 us; error 61 us; skew 500 ppm
Jun 20 11:33:20 dist-test-slave-ctbw krb5kdc[4707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955200, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:20.863412 4824 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:33:20.863875 4824 webserver.cc:460] Webserver started at http://127.2.243.129:35161/ using document root <none> and password file <none>
I20260620 11:33:20.864106 4824 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:20.864180 4824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:20.864328 4824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:20.865432 4824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "15d99a934eee48eca6d992034c546031"
format_stamp: "Formatted at 2026-06-20 11:33:20 on dist-test-slave-ctbw"
server_key: "9d969d55b37b82099a847326cbf77d8a"
server_key_iv: "521c159b3b4a42f159511d8bbd46559c"
server_key_version: "encryptionkey@0"
I20260620 11:33:20.865779 4824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "15d99a934eee48eca6d992034c546031"
format_stamp: "Formatted at 2026-06-20 11:33:20 on dist-test-slave-ctbw"
server_key: "9d969d55b37b82099a847326cbf77d8a"
server_key_iv: "521c159b3b4a42f159511d8bbd46559c"
server_key_version: "encryptionkey@0"
I20260620 11:33:20.867231 4824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 11:33:20.867997 4839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:20.868221 4824 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:20.868322 4824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "15d99a934eee48eca6d992034c546031"
format_stamp: "Formatted at 2026-06-20 11:33:20 on dist-test-slave-ctbw"
server_key: "9d969d55b37b82099a847326cbf77d8a"
server_key_iv: "521c159b3b4a42f159511d8bbd46559c"
server_key_version: "encryptionkey@0"
I20260620 11:33:20.868396 4824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:20.882988 4824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:20.884837 4824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:20.884945 4824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:20.885383 4824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:20.885449 4824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:20.885496 4824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:20.885521 4824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:20.892421 4824 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:44715
Jun 20 11:33:20 dist-test-slave-ctbw krb5kdc[4707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955200, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:20.892485 4954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:44715 every 8 connection(s)
I20260620 11:33:20.892817 4824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:33:20.893262 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 4824
I20260620 11:33:20.893371 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:33:20.893543 3022 external_mini_cluster.cc:1428] Setting key b7bcb77f9951a823b0ae590ce1dd57a0
I20260620 11:33:20.896229 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:20.885914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:49615 (local address 127.2.243.190:34259)
0620 11:33:20.885960 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:20.885962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:20.886178 (+ 216us) server_negotiation.cc:407] Connection header received
0620 11:33:20.886650 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:20.886651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:20.886664 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:20.886745 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:20.887578 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:20.888423 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:20.890004 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:20.890376 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:20.890463 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:20.893011 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:20.893013 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:20.893013 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:20.893021 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:20.894673 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:20.895167 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:20.895168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:20.895168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:20.895217 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:20.895501 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:20.895501 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:20.895502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:20.895604 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:20.895662 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:20.896004 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 11:33:20.896099 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:33:20.896253 4848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:20.885918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:34259 (local address 127.2.243.129:49615)
0620 11:33:20.886069 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:20.886102 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:20.886546 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:20.886820 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:20.886825 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:20.887184 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:20.887476 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:20.887482 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:20.888554 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:20.888556 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:20.889893 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:20.889896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:20.890476 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:20.890477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:20.890631 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:20.891173 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:20.891183 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:20.892901 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:20.894747 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:20.894753 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:20.894755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:20.895094 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:20.895287 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:20.895288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:20.895288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:20.895429 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:20.895667 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:20.895672 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:20.895837 (+ 165us) client_negotiation.cc:769] Sending connection context
0620 11:33:20.895950 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 11:33:20.896088 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260620 11:33:20.896800 4955 heartbeater.cc:347] Connected to a master server at 127.2.243.190:34259
I20260620 11:33:20.896893 4955 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:20.897084 4955 heartbeater.cc:511] Master 127.2.243.190:34259 requested a full tablet report, sending...
I20260620 11:33:20.897619 4751 ts_manager.cc:194] Registered new tserver with Master: 15d99a934eee48eca6d992034c546031 (127.2.243.129:44715)
I20260620 11:33:20.898234 4751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:49615
WARNING: no policy specified for kudu/127.2.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.130@KRBTEST.COM" created.
I20260620 11:33:20.904696 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:20.899072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:34259 (local address 127.2.243.129:45193)
0620 11:33:20.899236 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:20.899246 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:20.899310 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:20.899429 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:20.899429 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:20.899603 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:20.899673 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:20.899675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:20.900313 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:20.900314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:20.900819 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:20.900820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:20.901188 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:20.901189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:20.901261 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:20.901705 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:20.901709 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:20.902002 (+ 293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:20.903606 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:20.903606 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:20.903607 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:20.903918 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:20.904094 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:20.904094 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:20.904095 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:20.904152 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:20.904343 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:20.904344 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:20.904402 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 11:33:20.904455 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 11:33:20.904558 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":104,"threads_started":1}
I20260620 11:33:20.904769 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:20.899167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:45193 (local address 127.2.243.190:34259)
0620 11:33:20.899209 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:20.899211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:20.899305 (+ 94us) server_negotiation.cc:407] Connection header received
0620 11:33:20.899367 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:20.899368 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:20.899378 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:20.899426 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:20.899752 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:20.900220 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:20.900901 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:20.901101 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:20.901179 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:20.902115 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:20.902117 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:20.902117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:20.902124 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:20.903531 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:20.903986 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:20.903986 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:20.903987 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:20.904034 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:20.904205 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:20.904205 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:20.904205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:20.904297 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:20.904369 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:20.904502 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 11:33:20.904587 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
I20260620 11:33:20.919412 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:34259
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:21.000618 4964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:21.000824 4964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:21.000849 4964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:21.001915 4964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:21.001950 4964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:21.001969 4964 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 11:33:21.001982 4964 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 11:33:21.001998 4964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:33:21.003432 4964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.243.190:34259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:21.003640 4964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:21.003818 4964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:21.008657 4972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:21.008662 4969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:21.008857 4964 server_base.cc:1034] running on GCE node
W20260620 11:33:21.008677 4970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:21.009145 4964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:21.009405 4964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:21.010571 4964 hybrid_clock.cc:648] HybridClock initialized: now 1781955201010560 us; error 33 us; skew 500 ppm
Jun 20 11:33:21 dist-test-slave-ctbw krb5kdc[4707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955201, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:21.013245 4964 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:33:21.013726 4964 webserver.cc:460] Webserver started at http://127.2.243.130:45211/ using document root <none> and password file <none>
I20260620 11:33:21.013948 4964 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:21.014014 4964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:21.014171 4964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:21.015337 4964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "c03da774dea24698856551d74863f322"
format_stamp: "Formatted at 2026-06-20 11:33:21 on dist-test-slave-ctbw"
server_key: "6d3643f0daf09f7f18a7568c1eb80f87"
server_key_iv: "1ec78b30f81bf998f333cefa26a6cb45"
server_key_version: "encryptionkey@0"
I20260620 11:33:21.015702 4964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "c03da774dea24698856551d74863f322"
format_stamp: "Formatted at 2026-06-20 11:33:21 on dist-test-slave-ctbw"
server_key: "6d3643f0daf09f7f18a7568c1eb80f87"
server_key_iv: "1ec78b30f81bf998f333cefa26a6cb45"
server_key_version: "encryptionkey@0"
I20260620 11:33:21.017043 4964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:33:21.017855 4979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:21.018041 4964 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:21.018139 4964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "c03da774dea24698856551d74863f322"
format_stamp: "Formatted at 2026-06-20 11:33:21 on dist-test-slave-ctbw"
server_key: "6d3643f0daf09f7f18a7568c1eb80f87"
server_key_iv: "1ec78b30f81bf998f333cefa26a6cb45"
server_key_version: "encryptionkey@0"
I20260620 11:33:21.018222 4964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:21.033931 4964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:21.036464 4964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:21.036623 4964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:21.037196 4964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:21.037235 4964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:21.037271 4964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:21.037284 4964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:21.044512 4964 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:42187
I20260620 11:33:21.044570 5094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:42187 every 8 connection(s)
Jun 20 11:33:21 dist-test-slave-ctbw krb5kdc[4707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955201, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:21.044924 4964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:33:21.048204 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.037867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:52693 (local address 127.2.243.190:34259)
0620 11:33:21.037926 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:21.037928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:21.038138 (+ 210us) server_negotiation.cc:407] Connection header received
0620 11:33:21.038718 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:21.038719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:21.038732 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:21.038777 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:21.039693 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.040552 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.042141 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.042438 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.042503 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.045538 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:21.045539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:21.045540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:21.045547 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:21.046846 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:21.047311 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:21.047312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:21.047312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:21.047367 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:21.047635 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:21.047635 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:21.047636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:21.047742 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:21.047794 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:21.048069 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 11:33:21.048140 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 11:33:21.048427 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.037831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:34259 (local address 127.2.243.130:52693)
0620 11:33:21.038017 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:21.038053 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:21.038600 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:21.038882 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:21.038888 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:21.039261 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:21.039544 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.039552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.040661 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.040663 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.042024 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.042027 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.042543 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.042545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.042755 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.043436 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:21.043447 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:21.045426 (+ 1979us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:21.046993 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:21.046999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:21.047000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:21.047238 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:21.047437 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:21.047438 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:21.047438 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:21.047541 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:21.047802 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:21.047806 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:21.047953 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 11:33:21.048044 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 11:33:21.048170 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
I20260620 11:33:21.049018 5095 heartbeater.cc:347] Connected to a master server at 127.2.243.190:34259
I20260620 11:33:21.049119 5095 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:21.049316 5095 heartbeater.cc:511] Master 127.2.243.190:34259 requested a full tablet report, sending...
I20260620 11:33:21.049690 4752 ts_manager.cc:194] Registered new tserver with Master: c03da774dea24698856551d74863f322 (127.2.243.130:42187)
I20260620 11:33:21.050077 4752 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:52693
I20260620 11:33:21.054339 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 4964
I20260620 11:33:21.054459 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:33:21.054672 3022 external_mini_cluster.cc:1428] Setting key 471c69daf0dab555328d7ca6349225ad
I20260620 11:33:21.056361 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.050314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:44699 (local address 127.2.243.190:34259)
0620 11:33:21.050356 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:21.050358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:21.050384 (+ 26us) server_negotiation.cc:407] Connection header received
0620 11:33:21.050405 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:21.050406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:21.050415 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:21.050465 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:21.050819 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.051363 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.052117 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.052353 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.052450 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.053466 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:21.053467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:21.053467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:21.053474 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:21.055036 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:21.055412 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:21.055414 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:21.055415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:21.055471 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:21.055725 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:21.055726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:21.055726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:21.055879 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:21.055984 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:21.056180 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 11:33:21.056258 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:33:21.056377 5101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.050162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:34259 (local address 127.2.243.130:44699)
0620 11:33:21.050281 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:21.050294 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:21.050349 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:21.050468 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:21.050469 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:21.050666 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:21.050725 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.050726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.051458 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.051459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.052013 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.052015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.052452 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.052452 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.052538 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.053019 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:21.053024 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:21.053350 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:21.055130 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:21.055131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:21.055132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:21.055312 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:21.055582 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:21.055583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:21.055584 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:21.055652 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:21.055976 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:21.055977 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:21.056038 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 11:33:21.056147 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 11:33:21.056259 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for kudu/127.2.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
I20260620 11:33:21.081095 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:34259
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:33309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:21.162060 5104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:21.162272 5104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:21.162299 5104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:21.163543 5104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:21.163592 5104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:21.163614 5104 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 11:33:21.163627 5104 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 11:33:21.163703 5104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:33:21.165171 5104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:33309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.243.190:34259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:21.165434 5104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:21.165704 5104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:21.170946 5109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:21.171136 5110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:21.170976 5112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:21.171530 5104 server_base.cc:1034] running on GCE node
I20260620 11:33:21.171737 5104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:21.172001 5104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:21.173180 5104 hybrid_clock.cc:648] HybridClock initialized: now 1781955201173173 us; error 36 us; skew 500 ppm
Jun 20 11:33:21 dist-test-slave-ctbw krb5kdc[4707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955201, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:21.175880 5104 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:33:21.176365 5104 webserver.cc:460] Webserver started at http://127.2.243.131:39153/ using document root <none> and password file <none>
I20260620 11:33:21.176590 5104 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:21.176666 5104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:21.176815 5104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:21.177867 5104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "00a246165e5246ec8680e4d4ef91dcf9"
format_stamp: "Formatted at 2026-06-20 11:33:21 on dist-test-slave-ctbw"
server_key: "bedecd7f43b37167114a4757796b2d73"
server_key_iv: "4ca10779b7289aea099ce5eb53bec9c6"
server_key_version: "encryptionkey@0"
I20260620 11:33:21.178232 5104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "00a246165e5246ec8680e4d4ef91dcf9"
format_stamp: "Formatted at 2026-06-20 11:33:21 on dist-test-slave-ctbw"
server_key: "bedecd7f43b37167114a4757796b2d73"
server_key_iv: "4ca10779b7289aea099ce5eb53bec9c6"
server_key_version: "encryptionkey@0"
I20260620 11:33:21.179700 5104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:33:21.180469 5119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:21.180707 5104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:21.180814 5104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "00a246165e5246ec8680e4d4ef91dcf9"
format_stamp: "Formatted at 2026-06-20 11:33:21 on dist-test-slave-ctbw"
server_key: "bedecd7f43b37167114a4757796b2d73"
server_key_iv: "4ca10779b7289aea099ce5eb53bec9c6"
server_key_version: "encryptionkey@0"
I20260620 11:33:21.180887 5104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:21.205669 5104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:21.208204 5104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:21.208369 5104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:21.209079 5104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:21.209151 5104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:21.209184 5104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:21.209213 5104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 11:33:21 dist-test-slave-ctbw krb5kdc[4707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955201, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:21.216409 5104 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:46595
I20260620 11:33:21.216500 5234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:46595 every 8 connection(s)
I20260620 11:33:21.216820 5104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:33:21.219388 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.209472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:37537 (local address 127.2.243.190:34259)
0620 11:33:21.209518 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:21.209520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:21.209828 (+ 308us) server_negotiation.cc:407] Connection header received
0620 11:33:21.210383 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:21.210384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:21.210396 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:21.210448 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:21.211190 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.211822 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.213245 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.213466 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.213557 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.216210 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:21.216212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:21.216212 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:21.216220 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:21.217888 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:21.218341 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:21.218342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:21.218342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:21.218412 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:21.218676 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:21.218678 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:21.218678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:21.218805 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:21.218901 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:21.219207 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 11:33:21.219293 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:33:21.219563 5128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.209503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:34259 (local address 127.2.243.131:37537)
0620 11:33:21.209686 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:21.209729 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:21.210271 (+ 542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:21.210526 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:21.210530 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:21.210831 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:21.211089 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.211094 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.211919 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.211920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.213115 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.213118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.213580 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.213581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.213717 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.214301 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:21.214314 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:21.216105 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:21.217993 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:21.218000 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:21.218002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:21.218241 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:21.218482 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:21.218483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:21.218484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:21.218604 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:21.218897 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:21.218904 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:21.219052 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 11:33:21.219173 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 11:33:21.219332 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":120,"threads_started":1}
I20260620 11:33:21.220080 5235 heartbeater.cc:347] Connected to a master server at 127.2.243.190:34259
I20260620 11:33:21.220196 5235 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:21.220367 5235 heartbeater.cc:511] Master 127.2.243.190:34259 requested a full tablet report, sending...
I20260620 11:33:21.220819 4751 ts_manager.cc:194] Registered new tserver with Master: 00a246165e5246ec8680e4d4ef91dcf9 (127.2.243.131:46595)
I20260620 11:33:21.221359 4751 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:37537
I20260620 11:33:21.226331 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 5104
I20260620 11:33:21.226437 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:33:21.226634 3022 external_mini_cluster.cc:1428] Setting key 94f4e75569995b4d3b606d7d53410759
I20260620 11:33:21.227578 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.221188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:34259 (local address 127.2.243.131:47509)
0620 11:33:21.221381 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:21.221399 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:21.221483 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:21.222099 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:21.222100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:21.222252 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:21.222306 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.222307 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.222979 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.222980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.223500 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.223501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.223874 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.223875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.223944 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.224430 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:21.224436 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:21.224768 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:21.226105 (+ 1337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:21.226106 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:21.226106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:21.226353 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:21.226680 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:21.226681 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:21.226681 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:21.226755 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:21.227210 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:21.227211 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:21.227270 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 11:33:21.227357 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 11:33:21.227450 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":74,"threads_started":1}
I20260620 11:33:21.227592 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.221245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:47509 (local address 127.2.243.190:34259)
0620 11:33:21.221290 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:21.221292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:21.221469 (+ 177us) server_negotiation.cc:407] Connection header received
0620 11:33:21.221569 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:21.221570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:21.221581 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:21.221640 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:21.222380 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.222842 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.223567 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.223758 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.223814 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.224842 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:21.224843 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:21.224843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:21.224848 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:21.226006 (+ 1158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:21.226464 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:21.226465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:21.226465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:21.226546 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:21.226911 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:21.226912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:21.226912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:21.227031 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:21.227147 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:21.227403 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 11:33:21.227487 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:33:21.227766 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 11:33:21 dist-test-slave-ctbw krb5kdc[4707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 11:33:21 dist-test-slave-ctbw krb5kdc[4707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:21.251313 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.244375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37284 (local address 127.2.243.190:34259)
0620 11:33:21.244437 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:21.244439 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:21.244450 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:33:21.244530 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:21.244531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:21.244540 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:21.244598 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:21.245127 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.245683 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.246392 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.246585 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.246644 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.248896 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:21.248898 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:21.248898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:21.248904 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:21.250270 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:21.250591 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:21.250591 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:21.250592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:21.250640 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:21.250840 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:21.250840 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:21.250840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:21.250993 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:21.251069 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:21.251190 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 11:33:21.251249 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 11:33:21.253044 4751 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:33:21.254603 4751 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:33:21.261704 5252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.257179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:42187 (local address 127.0.0.1:41014)
0620 11:33:21.257443 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:21.257493 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:21.257562 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:21.258052 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:21.258053 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:21.258055 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:21.258108 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.258110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.259123 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.259124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.260399 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.260402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.261401 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.261402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.261469 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.261473 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:33:21.261550 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 11:33:21.261597 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"mutex_wait_us":1,"thread_start_us":64,"threads_started":1}
I20260620 11:33:21.261734 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.257417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41014 (local address 127.2.243.130:42187)
0620 11:33:21.257799 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:21.257803 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:21.257838 (+ 35us) server_negotiation.cc:407] Connection header received
0620 11:33:21.257877 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:21.257880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:21.257896 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:21.257990 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:21.258180 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.259037 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.260492 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.261314 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.261402 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.261467 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:21.261568 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 11:33:21.261609 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":97,"threads_started":1}
I20260620 11:33:21.261919 5257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.257619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38372 (local address 127.2.243.129:44715)
0620 11:33:21.257769 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:21.257773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:21.257781 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:33:21.257833 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:21.257834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:21.257846 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:21.257934 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:21.258120 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.259412 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.260676 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.261277 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.261651 (+ 374us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.261710 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:21.261736 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 11:33:21.261784 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260620 11:33:21.261965 5253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.257179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:44715 (local address 127.0.0.1:38372)
0620 11:33:21.257454 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:21.257493 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:21.257560 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:21.257941 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:21.257945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:21.257947 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:21.258021 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.258024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.259502 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.259503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.260596 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.260597 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.261376 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.261377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.261458 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.261473 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 11:33:21.261519 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 11:33:21.261570 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":5,"thread_start_us":75,"threads_started":1}
I20260620 11:33:21.262472 5010 tablet_service.cc:1467] Processing CreateTablet for tablet 41ddff224cb449f2a07a8a3a5de1da49 (DEFAULT_TABLE table=test-table [id=bfe745f5f9a7480da821be6c9a23e97b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:21.262856 5254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.257289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:46595 (local address 127.0.0.1:37588)
0620 11:33:21.258209 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:21.258219 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:21.258260 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:21.258535 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:21.258536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:21.258539 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:21.258579 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.258580 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.259823 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.259824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.260959 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.260961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.262556 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.262557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.262618 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.262622 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:33:21.262740 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 11:33:21.262778 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":902,"thread_start_us":82,"threads_started":1}
I20260620 11:33:21.262893 5010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ddff224cb449f2a07a8a3a5de1da49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:21.262931 5255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.257399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37588 (local address 127.2.243.131:46595)
0620 11:33:21.258369 (+ 970us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:21.258373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:21.258382 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:33:21.258418 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:21.258420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:21.258435 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:21.258522 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:21.258668 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.259735 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.261050 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.261740 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.262470 (+ 730us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.262530 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:21.262756 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 11:33:21.262805 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":937,"thread_start_us":82,"threads_started":1}
I20260620 11:33:21.262801 4889 tablet_service.cc:1467] Processing CreateTablet for tablet 41ddff224cb449f2a07a8a3a5de1da49 (DEFAULT_TABLE table=test-table [id=bfe745f5f9a7480da821be6c9a23e97b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:21.263178 4889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ddff224cb449f2a07a8a3a5de1da49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:21.263779 5169 tablet_service.cc:1467] Processing CreateTablet for tablet 41ddff224cb449f2a07a8a3a5de1da49 (DEFAULT_TABLE table=test-table [id=bfe745f5f9a7480da821be6c9a23e97b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:21.264086 5169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ddff224cb449f2a07a8a3a5de1da49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:21.266453 5258 tablet_bootstrap.cc:492] T 41ddff224cb449f2a07a8a3a5de1da49 P 15d99a934eee48eca6d992034c546031: Bootstrap starting.
I20260620 11:33:21.267160 5259 tablet_bootstrap.cc:492] T 41ddff224cb449f2a07a8a3a5de1da49 P c03da774dea24698856551d74863f322: Bootstrap starting.
I20260620 11:33:21.267199 5260 tablet_bootstrap.cc:492] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9: Bootstrap starting.
I20260620 11:33:21.267315 5258 tablet_bootstrap.cc:654] T 41ddff224cb449f2a07a8a3a5de1da49 P 15d99a934eee48eca6d992034c546031: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:21.267717 5258 log.cc:824] T 41ddff224cb449f2a07a8a3a5de1da49 P 15d99a934eee48eca6d992034c546031: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:21.267992 5259 tablet_bootstrap.cc:654] T 41ddff224cb449f2a07a8a3a5de1da49 P c03da774dea24698856551d74863f322: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:21.268225 5260 tablet_bootstrap.cc:654] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:21.268419 5259 log.cc:824] T 41ddff224cb449f2a07a8a3a5de1da49 P c03da774dea24698856551d74863f322: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:21.268563 5260 log.cc:824] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:21.268823 5258 tablet_bootstrap.cc:492] T 41ddff224cb449f2a07a8a3a5de1da49 P 15d99a934eee48eca6d992034c546031: No bootstrap required, opened a new log
I20260620 11:33:21.268896 5258 ts_tablet_manager.cc:1397] T 41ddff224cb449f2a07a8a3a5de1da49 P 15d99a934eee48eca6d992034c546031: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 11:33:21.269347 5260 tablet_bootstrap.cc:492] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9: No bootstrap required, opened a new log
I20260620 11:33:21.269429 5260 ts_tablet_manager.cc:1397] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:33:21.271129 5258 raft_consensus.cc:348] T 41ddff224cb449f2a07a8a3a5de1da49 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.271296 5259 tablet_bootstrap.cc:492] T 41ddff224cb449f2a07a8a3a5de1da49 P c03da774dea24698856551d74863f322: No bootstrap required, opened a new log
I20260620 11:33:21.271312 5258 raft_consensus.cc:374] T 41ddff224cb449f2a07a8a3a5de1da49 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:21.271381 5258 raft_consensus.cc:729] T 41ddff224cb449f2a07a8a3a5de1da49 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d99a934eee48eca6d992034c546031, State: Initialized, Role: FOLLOWER
I20260620 11:33:21.271409 5259 ts_tablet_manager.cc:1397] T 41ddff224cb449f2a07a8a3a5de1da49 P c03da774dea24698856551d74863f322: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 11:33:21.271528 5258 consensus_queue.cc:260] T 41ddff224cb449f2a07a8a3a5de1da49 P 15d99a934eee48eca6d992034c546031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.271588 5260 raft_consensus.cc:348] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.271759 5260 raft_consensus.cc:374] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:21.271809 5260 raft_consensus.cc:729] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a246165e5246ec8680e4d4ef91dcf9, State: Initialized, Role: FOLLOWER
I20260620 11:33:21.271849 5258 ts_tablet_manager.cc:1428] T 41ddff224cb449f2a07a8a3a5de1da49 P 15d99a934eee48eca6d992034c546031: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:33:21.271934 5260 consensus_queue.cc:260] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.272007 4955 heartbeater.cc:503] Master 127.2.243.190:34259 was elected leader, sending a full tablet report...
I20260620 11:33:21.272225 5260 ts_tablet_manager.cc:1428] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:33:21.272235 5235 heartbeater.cc:503] Master 127.2.243.190:34259 was elected leader, sending a full tablet report...
I20260620 11:33:21.273463 5259 raft_consensus.cc:348] T 41ddff224cb449f2a07a8a3a5de1da49 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.273581 5259 raft_consensus.cc:374] T 41ddff224cb449f2a07a8a3a5de1da49 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:21.273599 5259 raft_consensus.cc:729] T 41ddff224cb449f2a07a8a3a5de1da49 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03da774dea24698856551d74863f322, State: Initialized, Role: FOLLOWER
I20260620 11:33:21.273697 5259 consensus_queue.cc:260] T 41ddff224cb449f2a07a8a3a5de1da49 P c03da774dea24698856551d74863f322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.273903 5259 ts_tablet_manager.cc:1428] T 41ddff224cb449f2a07a8a3a5de1da49 P c03da774dea24698856551d74863f322: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 11:33:21.273975 5095 heartbeater.cc:503] Master 127.2.243.190:34259 was elected leader, sending a full tablet report...
I20260620 11:33:21.289419 5265 raft_consensus.cc:482] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:33:21.289544 5265 raft_consensus.cc:504] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.290012 5265 leader_election.cc:288] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c03da774dea24698856551d74863f322 (127.2.243.130:42187), 15d99a934eee48eca6d992034c546031 (127.2.243.129:44715)
I20260620 11:33:21.293591 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.290351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:41535 (local address 127.2.243.130:42187)
0620 11:33:21.290396 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:21.290398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:21.290407 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:33:21.290436 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:21.290437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:21.290455 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:21.290545 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:21.290715 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.291330 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.292854 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.293218 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.293271 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.293279 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:21.293469 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 11:33:21.293516 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:33:21.293614 5267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.290245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:44715 (local address 127.2.243.131:54257)
0620 11:33:21.290417 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:21.290426 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:21.290501 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:21.290652 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:21.290653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:21.290666 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:21.290729 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.290730 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.291439 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.291441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.292741 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.292744 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.293338 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.293339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.293413 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.293416 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:33:21.293491 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 11:33:21.293524 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":41,"thread_start_us":100,"threads_started":1}
I20260620 11:33:21.293615 5257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.290290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:54257 (local address 127.2.243.129:44715)
0620 11:33:21.290344 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:21.290346 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:21.290493 (+ 147us) server_negotiation.cc:407] Connection header received
0620 11:33:21.290565 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:21.290566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:21.290574 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:21.290637 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:21.290807 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.291352 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.292852 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.293222 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.293324 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.293332 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:21.293509 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 11:33:21.293550 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:33:21.293965 4909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ddff224cb449f2a07a8a3a5de1da49" candidate_uuid: "00a246165e5246ec8680e4d4ef91dcf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d99a934eee48eca6d992034c546031" is_pre_election: true
I20260620 11:33:21.293617 5128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.290187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:42187 (local address 127.2.243.131:41535)
0620 11:33:21.290287 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:21.290299 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:21.290351 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:21.290553 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:21.290554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:21.290556 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:21.290634 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.290636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.291439 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.291440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.292761 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.292763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.293302 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.293303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.293382 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.293386 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:33:21.293443 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 11:33:21.293503 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":49}
I20260620 11:33:21.294118 4909 raft_consensus.cc:2454] T 41ddff224cb449f2a07a8a3a5de1da49 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a246165e5246ec8680e4d4ef91dcf9 in term 0.
I20260620 11:33:21.294376 5122 leader_election.cc:302] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a246165e5246ec8680e4d4ef91dcf9, 15d99a934eee48eca6d992034c546031; no voters:
I20260620 11:33:21.294397 5049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ddff224cb449f2a07a8a3a5de1da49" candidate_uuid: "00a246165e5246ec8680e4d4ef91dcf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03da774dea24698856551d74863f322" is_pre_election: true
I20260620 11:33:21.294497 5265 raft_consensus.cc:2781] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:33:21.294553 5265 raft_consensus.cc:482] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:33:21.294586 5265 raft_consensus.cc:3037] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:21.294567 5049 raft_consensus.cc:2454] T 41ddff224cb449f2a07a8a3a5de1da49 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00a246165e5246ec8680e4d4ef91dcf9 in term 0.
I20260620 11:33:21.295361 5265 raft_consensus.cc:504] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
W20260620 11:33:21.295456 5096 tablet.cc:2369] T 41ddff224cb449f2a07a8a3a5de1da49 P c03da774dea24698856551d74863f322: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:33:21.295504 5265 leader_election.cc:288] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9 [CANDIDATE]: Term 1 election: Requested vote from peers c03da774dea24698856551d74863f322 (127.2.243.130:42187), 15d99a934eee48eca6d992034c546031 (127.2.243.129:44715)
I20260620 11:33:21.295706 5049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ddff224cb449f2a07a8a3a5de1da49" candidate_uuid: "00a246165e5246ec8680e4d4ef91dcf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03da774dea24698856551d74863f322"
I20260620 11:33:21.295742 4909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41ddff224cb449f2a07a8a3a5de1da49" candidate_uuid: "00a246165e5246ec8680e4d4ef91dcf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d99a934eee48eca6d992034c546031"
I20260620 11:33:21.295823 5049 raft_consensus.cc:3037] T 41ddff224cb449f2a07a8a3a5de1da49 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:21.295823 4909 raft_consensus.cc:3037] T 41ddff224cb449f2a07a8a3a5de1da49 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:21.296551 5049 raft_consensus.cc:2454] T 41ddff224cb449f2a07a8a3a5de1da49 P c03da774dea24698856551d74863f322 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a246165e5246ec8680e4d4ef91dcf9 in term 1.
I20260620 11:33:21.296730 4909 raft_consensus.cc:2454] T 41ddff224cb449f2a07a8a3a5de1da49 P 15d99a934eee48eca6d992034c546031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00a246165e5246ec8680e4d4ef91dcf9 in term 1.
I20260620 11:33:21.296748 5120 leader_election.cc:302] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a246165e5246ec8680e4d4ef91dcf9, c03da774dea24698856551d74863f322; no voters:
I20260620 11:33:21.296885 5265 raft_consensus.cc:2781] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:21.297056 5265 raft_consensus.cc:686] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9 [term 1 LEADER]: Becoming Leader. State: Replica: 00a246165e5246ec8680e4d4ef91dcf9, State: Running, Role: LEADER
I20260620 11:33:21.297163 5265 consensus_queue.cc:237] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.297959 4752 catalog_manager.cc:5495] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00a246165e5246ec8680e4d4ef91dcf9 (127.2.243.131). New cstate: current_term: 1 leader_uuid: "00a246165e5246ec8680e4d4ef91dcf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } health_report { overall_health: HEALTHY } } }
Jun 20 11:33:21 dist-test-slave-ctbw krb5kdc[4707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955200, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:21.327425 4801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.319676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37296 (local address 127.2.243.190:34259)
0620 11:33:21.319751 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:21.319754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:21.319908 (+ 154us) server_negotiation.cc:407] Connection header received
0620 11:33:21.320008 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:21.320009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:21.320020 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:21.320103 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:21.320671 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.321355 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.322225 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.322437 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.322527 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.324774 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:21.324776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:21.324776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:21.324783 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:21.326192 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:21.326599 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:21.326600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:21.326600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:21.326665 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:21.326854 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:21.326855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:21.326855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:21.326998 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:21.327074 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:21.327228 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 11:33:21.327312 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 11:33:21.327502 5275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.319580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:34259 (local address 127.0.0.1:37296)
0620 11:33:21.319801 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:21.319812 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:21.319930 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:21.320111 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:21.320112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:21.320479 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:21.320534 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.320536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.321449 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.321450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.322102 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.322104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.322527 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.322528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.322585 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.323244 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:21.323249 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:21.324669 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:21.326295 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:21.326302 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:21.326304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:21.326515 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:21.326729 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:21.326729 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:21.326730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:21.326788 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:21.327063 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:21.327070 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:21.327114 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 11:33:21.327206 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 11:33:21.327370 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":67,"thread_start_us":116,"threads_started":1}
I20260620 11:33:21.329666 4752 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:37296:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 11:33:21.329874 4752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:33:21.331678 5010 tablet_service.cc:1467] Processing CreateTablet for tablet bb1a1e4466b7424bb42884cf04a59937 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e56c88ebc344d76a9bb539527322240]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 11:33:21.331660 4889 tablet_service.cc:1467] Processing CreateTablet for tablet bb1a1e4466b7424bb42884cf04a59937 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e56c88ebc344d76a9bb539527322240]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 11:33:21.331862 5010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb1a1e4466b7424bb42884cf04a59937. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:21.331892 4889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb1a1e4466b7424bb42884cf04a59937. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:21.331894 5169 tablet_service.cc:1467] Processing CreateTablet for tablet bb1a1e4466b7424bb42884cf04a59937 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e56c88ebc344d76a9bb539527322240]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 11:33:21.332032 5169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb1a1e4466b7424bb42884cf04a59937. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:21.333477 5258 tablet_bootstrap.cc:492] T bb1a1e4466b7424bb42884cf04a59937 P 15d99a934eee48eca6d992034c546031: Bootstrap starting.
I20260620 11:33:21.333477 5260 tablet_bootstrap.cc:492] T bb1a1e4466b7424bb42884cf04a59937 P 00a246165e5246ec8680e4d4ef91dcf9: Bootstrap starting.
I20260620 11:33:21.333492 5259 tablet_bootstrap.cc:492] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322: Bootstrap starting.
I20260620 11:33:21.333956 5260 tablet_bootstrap.cc:654] T bb1a1e4466b7424bb42884cf04a59937 P 00a246165e5246ec8680e4d4ef91dcf9: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:21.334147 5259 tablet_bootstrap.cc:654] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:21.334200 5258 tablet_bootstrap.cc:654] T bb1a1e4466b7424bb42884cf04a59937 P 15d99a934eee48eca6d992034c546031: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:21.334693 5260 tablet_bootstrap.cc:492] T bb1a1e4466b7424bb42884cf04a59937 P 00a246165e5246ec8680e4d4ef91dcf9: No bootstrap required, opened a new log
I20260620 11:33:21.334775 5260 ts_tablet_manager.cc:1397] T bb1a1e4466b7424bb42884cf04a59937 P 00a246165e5246ec8680e4d4ef91dcf9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:21.334977 5258 tablet_bootstrap.cc:492] T bb1a1e4466b7424bb42884cf04a59937 P 15d99a934eee48eca6d992034c546031: No bootstrap required, opened a new log
I20260620 11:33:21.334978 5260 raft_consensus.cc:348] T bb1a1e4466b7424bb42884cf04a59937 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.335027 5258 ts_tablet_manager.cc:1397] T bb1a1e4466b7424bb42884cf04a59937 P 15d99a934eee48eca6d992034c546031: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:33:21.335053 5260 raft_consensus.cc:374] T bb1a1e4466b7424bb42884cf04a59937 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:21.335078 5260 raft_consensus.cc:729] T bb1a1e4466b7424bb42884cf04a59937 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a246165e5246ec8680e4d4ef91dcf9, State: Initialized, Role: FOLLOWER
I20260620 11:33:21.335129 5260 consensus_queue.cc:260] T bb1a1e4466b7424bb42884cf04a59937 P 00a246165e5246ec8680e4d4ef91dcf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.335163 5258 raft_consensus.cc:348] T bb1a1e4466b7424bb42884cf04a59937 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.335220 5258 raft_consensus.cc:374] T bb1a1e4466b7424bb42884cf04a59937 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:21.335247 5258 raft_consensus.cc:729] T bb1a1e4466b7424bb42884cf04a59937 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d99a934eee48eca6d992034c546031, State: Initialized, Role: FOLLOWER
I20260620 11:33:21.335239 5260 ts_tablet_manager.cc:1428] T bb1a1e4466b7424bb42884cf04a59937 P 00a246165e5246ec8680e4d4ef91dcf9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:21.335290 5258 consensus_queue.cc:260] T bb1a1e4466b7424bb42884cf04a59937 P 15d99a934eee48eca6d992034c546031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.335287 5265 tablet_replica.cc:439] T bb1a1e4466b7424bb42884cf04a59937 P 00a246165e5246ec8680e4d4ef91dcf9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } } }
I20260620 11:33:21.335382 5265 tablet_replica.cc:442] T bb1a1e4466b7424bb42884cf04a59937 P 00a246165e5246ec8680e4d4ef91dcf9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:33:21.335391 5258 ts_tablet_manager.cc:1428] T bb1a1e4466b7424bb42884cf04a59937 P 15d99a934eee48eca6d992034c546031: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:21.335561 5264 tablet_replica.cc:439] T bb1a1e4466b7424bb42884cf04a59937 P 15d99a934eee48eca6d992034c546031: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } } }
I20260620 11:33:21.335616 5259 tablet_bootstrap.cc:492] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322: No bootstrap required, opened a new log
I20260620 11:33:21.335649 5264 tablet_replica.cc:442] T bb1a1e4466b7424bb42884cf04a59937 P 15d99a934eee48eca6d992034c546031: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:33:21.335705 5259 ts_tablet_manager.cc:1397] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:33:21.335848 5259 raft_consensus.cc:348] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.335918 5259 raft_consensus.cc:374] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:21.335940 5259 raft_consensus.cc:729] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03da774dea24698856551d74863f322, State: Initialized, Role: FOLLOWER
I20260620 11:33:21.335983 5259 consensus_queue.cc:260] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.336054 5259 ts_tablet_manager.cc:1428] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:21.336144 5266 tablet_replica.cc:439] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } } }
I20260620 11:33:21.336247 5266 tablet_replica.cc:442] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:33:21.386328 5266 raft_consensus.cc:482] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:33:21.386445 5266 raft_consensus.cc:504] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.386920 5266 leader_election.cc:288] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15d99a934eee48eca6d992034c546031 (127.2.243.129:44715), 00a246165e5246ec8680e4d4ef91dcf9 (127.2.243.131:46595)
I20260620 11:33:21.391198 5255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.387226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:51537 (local address 127.2.243.131:46595)
0620 11:33:21.387431 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:21.387433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:21.387443 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:33:21.387526 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:21.387527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:21.387587 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:21.387619 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:21.387869 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.388631 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.390082 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.390729 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.390822 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.390830 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:21.391081 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 11:33:21.391126 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260620 11:33:21.391201 5257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.387357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:40617 (local address 127.2.243.129:44715)
0620 11:33:21.387413 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:21.387415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:21.387424 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:33:21.387454 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:21.387455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:21.387465 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:21.387523 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:21.387722 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.388569 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.390082 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.390735 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.390822 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.390830 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:21.391081 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 11:33:21.391126 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 11:33:21.391249 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.387090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:44715 (local address 127.2.243.130:40617)
0620 11:33:21.387229 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:21.387241 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:21.387297 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:21.387580 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:21.387581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:21.387582 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:21.387657 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.387659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.388661 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.388662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.389969 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.389971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.390892 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.390894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.390998 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.391000 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:33:21.391063 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 11:33:21.391121 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":12}
I20260620 11:33:21.391258 5279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.387170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:46595 (local address 127.2.243.130:51537)
0620 11:33:21.387332 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:21.387343 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:21.387423 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:21.387717 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:21.387718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:21.387720 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:21.387794 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.387795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.388740 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.388741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.389969 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.389971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.390920 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.390921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.390995 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.390999 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:33:21.391063 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 11:33:21.391121 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":44,"thread_start_us":104,"threads_started":1}
I20260620 11:33:21.391499 4909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb1a1e4466b7424bb42884cf04a59937" candidate_uuid: "c03da774dea24698856551d74863f322" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d99a934eee48eca6d992034c546031" is_pre_election: true
I20260620 11:33:21.391592 4909 raft_consensus.cc:2454] T bb1a1e4466b7424bb42884cf04a59937 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c03da774dea24698856551d74863f322 in term 0.
I20260620 11:33:21.391619 5189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb1a1e4466b7424bb42884cf04a59937" candidate_uuid: "c03da774dea24698856551d74863f322" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a246165e5246ec8680e4d4ef91dcf9" is_pre_election: true
I20260620 11:33:21.391783 5189 raft_consensus.cc:2454] T bb1a1e4466b7424bb42884cf04a59937 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c03da774dea24698856551d74863f322 in term 0.
I20260620 11:33:21.391765 4982 leader_election.cc:302] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15d99a934eee48eca6d992034c546031, c03da774dea24698856551d74863f322; no voters:
I20260620 11:33:21.391943 5266 raft_consensus.cc:2781] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:33:21.391992 5266 raft_consensus.cc:482] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:33:21.392014 5266 raft_consensus.cc:3037] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:21.392725 5266 raft_consensus.cc:504] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.392863 5266 leader_election.cc:288] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322 [CANDIDATE]: Term 1 election: Requested vote from peers 15d99a934eee48eca6d992034c546031 (127.2.243.129:44715), 00a246165e5246ec8680e4d4ef91dcf9 (127.2.243.131:46595)
I20260620 11:33:21.393039 5189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb1a1e4466b7424bb42884cf04a59937" candidate_uuid: "c03da774dea24698856551d74863f322" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a246165e5246ec8680e4d4ef91dcf9"
I20260620 11:33:21.393028 4909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb1a1e4466b7424bb42884cf04a59937" candidate_uuid: "c03da774dea24698856551d74863f322" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d99a934eee48eca6d992034c546031"
I20260620 11:33:21.393110 4909 raft_consensus.cc:3037] T bb1a1e4466b7424bb42884cf04a59937 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:21.393110 5189 raft_consensus.cc:3037] T bb1a1e4466b7424bb42884cf04a59937 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Advancing to term 1
W20260620 11:33:21.393513 4956 tablet.cc:2369] T 41ddff224cb449f2a07a8a3a5de1da49 P 15d99a934eee48eca6d992034c546031: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:33:21.393879 4909 raft_consensus.cc:2454] T bb1a1e4466b7424bb42884cf04a59937 P 15d99a934eee48eca6d992034c546031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c03da774dea24698856551d74863f322 in term 1.
I20260620 11:33:21.393977 5189 raft_consensus.cc:2454] T bb1a1e4466b7424bb42884cf04a59937 P 00a246165e5246ec8680e4d4ef91dcf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c03da774dea24698856551d74863f322 in term 1.
I20260620 11:33:21.394049 4982 leader_election.cc:302] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15d99a934eee48eca6d992034c546031, c03da774dea24698856551d74863f322; no voters:
I20260620 11:33:21.394155 5266 raft_consensus.cc:2781] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:21.394321 5266 raft_consensus.cc:686] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322 [term 1 LEADER]: Becoming Leader. State: Replica: c03da774dea24698856551d74863f322, State: Running, Role: LEADER
I20260620 11:33:21.394423 5266 consensus_queue.cc:237] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.394835 5280 tablet_replica.cc:439] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322: TxnStatusTablet state changed. Reason: New leader c03da774dea24698856551d74863f322. Latest consensus state: current_term: 1 leader_uuid: "c03da774dea24698856551d74863f322" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } } }
I20260620 11:33:21.394991 5280 tablet_replica.cc:442] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322: This TxnStatusTablet replica's current role is: LEADER
I20260620 11:33:21.395237 4750 catalog_manager.cc:5495] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322 reported cstate change: term changed from 0 to 1, leader changed from <none> to c03da774dea24698856551d74863f322 (127.2.243.130). New cstate: current_term: 1 leader_uuid: "c03da774dea24698856551d74863f322" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } health_report { overall_health: UNKNOWN } } }
I20260620 11:33:21.395529 5282 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 11:33:21.395610 5282 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 11:33:21.410979 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.408507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.2.243.130:42187)
0620 11:33:21.408549 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:21.408551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:21.408563 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:33:21.408665 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:21.408666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:21.408683 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:21.408747 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:21.408854 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.409349 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.410102 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.410305 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.410390 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.410581 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:21.410687 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:21.410789 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:21.410816 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 11:33:21.410838 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:33:21.410977 5275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.408445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:42187 (local address 127.0.0.1:41020)
0620 11:33:21.408524 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:21.408537 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:21.408583 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:21.408751 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:21.408751 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:21.408752 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:33:21.408804 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.408806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.409436 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.409437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.410006 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.410008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.410391 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.410392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.410459 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.410479 (+ 20us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:33:21.410755 (+ 276us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:33:21.410764 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 11:33:21.410799 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 11:33:21.410849 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 11:33:21.413134 5189 raft_consensus.cc:1261] T bb1a1e4466b7424bb42884cf04a59937 P 00a246165e5246ec8680e4d4ef91dcf9 [term 1 FOLLOWER]: Refusing update from remote peer c03da774dea24698856551d74863f322: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:21.413172 4909 raft_consensus.cc:1261] T bb1a1e4466b7424bb42884cf04a59937 P 15d99a934eee48eca6d992034c546031 [term 1 FOLLOWER]: Refusing update from remote peer c03da774dea24698856551d74863f322: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:21.413455 5266 consensus_queue.cc:1035] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:21.413590 5280 consensus_queue.cc:1035] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:21.414621 5264 tablet_replica.cc:439] T bb1a1e4466b7424bb42884cf04a59937 P 15d99a934eee48eca6d992034c546031: TxnStatusTablet state changed. Reason: New leader c03da774dea24698856551d74863f322. Latest consensus state: current_term: 1 leader_uuid: "c03da774dea24698856551d74863f322" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } } }
I20260620 11:33:21.414727 5264 tablet_replica.cc:442] T bb1a1e4466b7424bb42884cf04a59937 P 15d99a934eee48eca6d992034c546031: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:33:21.415818 5265 tablet_replica.cc:439] T bb1a1e4466b7424bb42884cf04a59937 P 00a246165e5246ec8680e4d4ef91dcf9: TxnStatusTablet state changed. Reason: New leader c03da774dea24698856551d74863f322. Latest consensus state: current_term: 1 leader_uuid: "c03da774dea24698856551d74863f322" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } } }
I20260620 11:33:21.415905 5265 tablet_replica.cc:442] T bb1a1e4466b7424bb42884cf04a59937 P 00a246165e5246ec8680e4d4ef91dcf9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:33:21.416425 5280 tablet_replica.cc:439] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c03da774dea24698856551d74863f322" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } } }
I20260620 11:33:21.416515 5280 tablet_replica.cc:442] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322: This TxnStatusTablet replica's current role is: LEADER
I20260620 11:33:21.416566 5280 tablet_replica.cc:439] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c03da774dea24698856551d74863f322" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } } }
I20260620 11:33:21.416608 5280 tablet_replica.cc:442] T bb1a1e4466b7424bb42884cf04a59937 P c03da774dea24698856551d74863f322: This TxnStatusTablet replica's current role is: LEADER
I20260620 11:33:21.416651 5265 tablet_replica.cc:439] T bb1a1e4466b7424bb42884cf04a59937 P 00a246165e5246ec8680e4d4ef91dcf9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c03da774dea24698856551d74863f322" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } } }
I20260620 11:33:21.416787 5265 tablet_replica.cc:442] T bb1a1e4466b7424bb42884cf04a59937 P 00a246165e5246ec8680e4d4ef91dcf9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:33:21.416931 5286 mvcc.cc:205] Tried to move back new op lower bound from 7298888504985550848 to 7298888504911847424. Current Snapshot: MvccSnapshot[applied={T|T < 7298888504985550848}]
I20260620 11:33:21.417117 5288 mvcc.cc:205] Tried to move back new op lower bound from 7298888504985550848 to 7298888504911847424. Current Snapshot: MvccSnapshot[applied={T|T < 7298888504985550848}]
I20260620 11:33:21.417304 5264 tablet_replica.cc:439] T bb1a1e4466b7424bb42884cf04a59937 P 15d99a934eee48eca6d992034c546031: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c03da774dea24698856551d74863f322" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } } }
I20260620 11:33:21.417402 5264 tablet_replica.cc:442] T bb1a1e4466b7424bb42884cf04a59937 P 15d99a934eee48eca6d992034c546031: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:33:21.417724 5285 mvcc.cc:205] Tried to move back new op lower bound from 7298888504985550848 to 7298888504911847424. Current Snapshot: MvccSnapshot[applied={T|T < 7298888504985550848}]
I20260620 11:33:21.422737 5255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.419774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37592 (local address 127.2.243.131:46595)
0620 11:33:21.419925 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:21.419927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:21.419933 (+ 6us) server_negotiation.cc:407] Connection header received
0620 11:33:21.419961 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:21.419962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:21.420000 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:21.420066 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:21.420249 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.421021 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.421905 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.422210 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.422285 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.422425 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:21.422541 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:21.422618 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:21.422639 (+ 21us) server_negotiation.cc:299] Negotiation successful
0620 11:33:21.422659 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260620 11:33:21.427619 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.425110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:33139 (local address 127.2.243.130:42187)
0620 11:33:21.425170 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:21.425172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:21.425239 (+ 67us) server_negotiation.cc:407] Connection header received
0620 11:33:21.425302 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:21.425302 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:21.425310 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:21.425348 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:21.425470 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.425911 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.426982 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.427172 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.427223 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.427411 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:21.427434 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:21.427474 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:21.427555 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 11:33:21.427570 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 11:33:21.427662 5241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.425049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:42187 (local address 127.2.243.131:33139)
0620 11:33:21.425179 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:21.425191 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:21.425244 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:21.425361 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:21.425361 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:21.425362 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:33:21.425416 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.425417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.425981 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.425982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.426913 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.426915 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.427240 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.427241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.427306 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.427320 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:33:21.427493 (+ 173us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:33:21.427495 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:33:21.427551 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 11:33:21.427590 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260620 11:33:21.430315 4909 raft_consensus.cc:1261] T 41ddff224cb449f2a07a8a3a5de1da49 P 15d99a934eee48eca6d992034c546031 [term 1 FOLLOWER]: Refusing update from remote peer 00a246165e5246ec8680e4d4ef91dcf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:21.430406 5049 raft_consensus.cc:1261] T 41ddff224cb449f2a07a8a3a5de1da49 P c03da774dea24698856551d74863f322 [term 1 FOLLOWER]: Refusing update from remote peer 00a246165e5246ec8680e4d4ef91dcf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:21.430555 5265 consensus_queue.cc:1035] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:21.430665 5268 consensus_queue.cc:1035] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:21.439612 5101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.436465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:46595 (local address 127.2.243.130:48817)
0620 11:33:21.436517 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:21.436529 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:21.436568 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:21.436785 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:21.436786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:21.436787 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:33:21.437057 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.437058 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.437938 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.437939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.438779 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.438782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.439256 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.439257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.439312 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.439326 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:33:21.439494 (+ 168us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:33:21.439496 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:33:21.439526 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 11:33:21.439559 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 11:33:21.439628 5255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.436642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:48817 (local address 127.2.243.131:46595)
0620 11:33:21.436687 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:21.436689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:21.436696 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:33:21.436715 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:21.436716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:21.436725 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:21.436781 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:21.437126 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.437860 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.438849 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.439178 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.439251 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.439403 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:21.439435 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:21.439488 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:21.439538 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 11:33:21.439557 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:33:21.449045 4750 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:21.449172 4750 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:21.449860 4750 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2: Sending DeleteTablet for 3 replicas of tablet 41ddff224cb449f2a07a8a3a5de1da49
I20260620 11:33:21.450205 5169 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ddff224cb449f2a07a8a3a5de1da49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:21 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:37588
I20260620 11:33:21.450205 4889 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ddff224cb449f2a07a8a3a5de1da49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:21 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:38372
I20260620 11:33:21.450536 5304 tablet_replica.cc:330] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9: stopping tablet replica
I20260620 11:33:21.450523 5010 tablet_service.cc:1514] Processing DeleteTablet for tablet 41ddff224cb449f2a07a8a3a5de1da49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:21 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:41014
I20260620 11:33:21.450634 5304 raft_consensus.cc:2229] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:33:21.450729 5304 raft_consensus.cc:2258] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:21.450904 5303 tablet_replica.cc:330] T 41ddff224cb449f2a07a8a3a5de1da49 P 15d99a934eee48eca6d992034c546031: stopping tablet replica
I20260620 11:33:21.451012 5303 raft_consensus.cc:2229] T 41ddff224cb449f2a07a8a3a5de1da49 P 15d99a934eee48eca6d992034c546031 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:21.451068 5303 raft_consensus.cc:2258] T 41ddff224cb449f2a07a8a3a5de1da49 P 15d99a934eee48eca6d992034c546031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:21.451063 4750 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37284:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:33:21.451211 4750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:33:21.451333 5304 ts_tablet_manager.cc:1905] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:21.451472 5303 ts_tablet_manager.cc:1905] T 41ddff224cb449f2a07a8a3a5de1da49 P 15d99a934eee48eca6d992034c546031: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:21.452384 4889 tablet_service.cc:1467] Processing CreateTablet for tablet fbfc0247cf764be3a74fd705b9e054c9 (DEFAULT_TABLE table=test-table [id=cdfbb7a920eb4264831cf1543210bdcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:21.452491 4889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbfc0247cf764be3a74fd705b9e054c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:21.453262 5010 tablet_service.cc:1467] Processing CreateTablet for tablet fbfc0247cf764be3a74fd705b9e054c9 (DEFAULT_TABLE table=test-table [id=cdfbb7a920eb4264831cf1543210bdcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:21.453384 5010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbfc0247cf764be3a74fd705b9e054c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:21.453917 5258 tablet_bootstrap.cc:492] T fbfc0247cf764be3a74fd705b9e054c9 P 15d99a934eee48eca6d992034c546031: Bootstrap starting.
I20260620 11:33:21.454283 5258 tablet_bootstrap.cc:654] T fbfc0247cf764be3a74fd705b9e054c9 P 15d99a934eee48eca6d992034c546031: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:21.454695 5303 ts_tablet_manager.cc:1918] T 41ddff224cb449f2a07a8a3a5de1da49 P 15d99a934eee48eca6d992034c546031: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:33:21.454741 5259 tablet_bootstrap.cc:492] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322: Bootstrap starting.
I20260620 11:33:21.454758 5303 log.cc:1196] T 41ddff224cb449f2a07a8a3a5de1da49 P 15d99a934eee48eca6d992034c546031: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-0/wal/wals/41ddff224cb449f2a07a8a3a5de1da49
I20260620 11:33:21.455072 5303 ts_tablet_manager.cc:1939] T 41ddff224cb449f2a07a8a3a5de1da49 P 15d99a934eee48eca6d992034c546031: Deleting consensus metadata
I20260620 11:33:21.455174 5259 tablet_bootstrap.cc:654] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:21.455183 5169 tablet_service.cc:1467] Processing CreateTablet for tablet fbfc0247cf764be3a74fd705b9e054c9 (DEFAULT_TABLE table=test-table [id=cdfbb7a920eb4264831cf1543210bdcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:21.455453 4738 catalog_manager.cc:4841] TS 15d99a934eee48eca6d992034c546031 (127.2.243.129:44715): tablet 41ddff224cb449f2a07a8a3a5de1da49 (table test-table [id=bfe745f5f9a7480da821be6c9a23e97b]) successfully deleted
I20260620 11:33:21.455543 5169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbfc0247cf764be3a74fd705b9e054c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:21.455781 5305 tablet_replica.cc:330] T 41ddff224cb449f2a07a8a3a5de1da49 P c03da774dea24698856551d74863f322: stopping tablet replica
I20260620 11:33:21.455884 5258 tablet_bootstrap.cc:492] T fbfc0247cf764be3a74fd705b9e054c9 P 15d99a934eee48eca6d992034c546031: No bootstrap required, opened a new log
I20260620 11:33:21.455932 5258 ts_tablet_manager.cc:1397] T fbfc0247cf764be3a74fd705b9e054c9 P 15d99a934eee48eca6d992034c546031: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 11:33:21.456027 5305 raft_consensus.cc:2229] T 41ddff224cb449f2a07a8a3a5de1da49 P c03da774dea24698856551d74863f322 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:21.456075 5258 raft_consensus.cc:348] T fbfc0247cf764be3a74fd705b9e054c9 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.456144 5258 raft_consensus.cc:374] T fbfc0247cf764be3a74fd705b9e054c9 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:21.456171 5258 raft_consensus.cc:729] T fbfc0247cf764be3a74fd705b9e054c9 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d99a934eee48eca6d992034c546031, State: Initialized, Role: FOLLOWER
I20260620 11:33:21.456213 5258 consensus_queue.cc:260] T fbfc0247cf764be3a74fd705b9e054c9 P 15d99a934eee48eca6d992034c546031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.456132 5305 raft_consensus.cc:2258] T 41ddff224cb449f2a07a8a3a5de1da49 P c03da774dea24698856551d74863f322 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:21.456306 5258 ts_tablet_manager.cc:1428] T fbfc0247cf764be3a74fd705b9e054c9 P 15d99a934eee48eca6d992034c546031: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 11:33:21.456821 5260 tablet_bootstrap.cc:492] T fbfc0247cf764be3a74fd705b9e054c9 P 00a246165e5246ec8680e4d4ef91dcf9: Bootstrap starting.
I20260620 11:33:21.456722 5305 ts_tablet_manager.cc:1905] T 41ddff224cb449f2a07a8a3a5de1da49 P c03da774dea24698856551d74863f322: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:21.456871 5304 ts_tablet_manager.cc:1918] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:33:21.456948 5304 log.cc:1196] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/41ddff224cb449f2a07a8a3a5de1da49
I20260620 11:33:21.457235 5260 tablet_bootstrap.cc:654] T fbfc0247cf764be3a74fd705b9e054c9 P 00a246165e5246ec8680e4d4ef91dcf9: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:21.457255 5304 ts_tablet_manager.cc:1939] T 41ddff224cb449f2a07a8a3a5de1da49 P 00a246165e5246ec8680e4d4ef91dcf9: Deleting consensus metadata
I20260620 11:33:21.457610 4737 catalog_manager.cc:4841] TS 00a246165e5246ec8680e4d4ef91dcf9 (127.2.243.131:46595): tablet 41ddff224cb449f2a07a8a3a5de1da49 (table test-table [id=bfe745f5f9a7480da821be6c9a23e97b]) successfully deleted
I20260620 11:33:21.458963 5260 tablet_bootstrap.cc:492] T fbfc0247cf764be3a74fd705b9e054c9 P 00a246165e5246ec8680e4d4ef91dcf9: No bootstrap required, opened a new log
I20260620 11:33:21.459017 5260 ts_tablet_manager.cc:1397] T fbfc0247cf764be3a74fd705b9e054c9 P 00a246165e5246ec8680e4d4ef91dcf9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:33:21.459158 5260 raft_consensus.cc:348] T fbfc0247cf764be3a74fd705b9e054c9 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.459224 5260 raft_consensus.cc:374] T fbfc0247cf764be3a74fd705b9e054c9 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:21.459250 5260 raft_consensus.cc:729] T fbfc0247cf764be3a74fd705b9e054c9 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a246165e5246ec8680e4d4ef91dcf9, State: Initialized, Role: FOLLOWER
I20260620 11:33:21.459291 5260 consensus_queue.cc:260] T fbfc0247cf764be3a74fd705b9e054c9 P 00a246165e5246ec8680e4d4ef91dcf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.459385 5260 ts_tablet_manager.cc:1428] T fbfc0247cf764be3a74fd705b9e054c9 P 00a246165e5246ec8680e4d4ef91dcf9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:21.459672 5259 tablet_bootstrap.cc:492] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322: No bootstrap required, opened a new log
I20260620 11:33:21.459717 5259 ts_tablet_manager.cc:1397] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 11:33:21.459936 5305 ts_tablet_manager.cc:1918] T 41ddff224cb449f2a07a8a3a5de1da49 P c03da774dea24698856551d74863f322: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:33:21.459988 5305 log.cc:1196] T 41ddff224cb449f2a07a8a3a5de1da49 P c03da774dea24698856551d74863f322: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-1/wal/wals/41ddff224cb449f2a07a8a3a5de1da49
I20260620 11:33:21.460011 5259 raft_consensus.cc:348] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.460086 5259 raft_consensus.cc:374] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:21.460111 5259 raft_consensus.cc:729] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03da774dea24698856551d74863f322, State: Initialized, Role: FOLLOWER
I20260620 11:33:21.460156 5259 consensus_queue.cc:260] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.460239 5305 ts_tablet_manager.cc:1939] T 41ddff224cb449f2a07a8a3a5de1da49 P c03da774dea24698856551d74863f322: Deleting consensus metadata
I20260620 11:33:21.460250 5259 ts_tablet_manager.cc:1428] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:21.460589 4736 catalog_manager.cc:4841] TS c03da774dea24698856551d74863f322 (127.2.243.130:42187): tablet 41ddff224cb449f2a07a8a3a5de1da49 (table test-table [id=bfe745f5f9a7480da821be6c9a23e97b]) successfully deleted
W20260620 11:33:21.467705 5236 tablet.cc:2369] T fbfc0247cf764be3a74fd705b9e054c9 P 00a246165e5246ec8680e4d4ef91dcf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:33:21.508586 5280 raft_consensus.cc:482] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:33:21.508705 5280 raft_consensus.cc:504] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.508852 5280 leader_election.cc:288] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15d99a934eee48eca6d992034c546031 (127.2.243.129:44715), 00a246165e5246ec8680e4d4ef91dcf9 (127.2.243.131:46595)
I20260620 11:33:21.509066 4909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbfc0247cf764be3a74fd705b9e054c9" candidate_uuid: "c03da774dea24698856551d74863f322" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d99a934eee48eca6d992034c546031" is_pre_election: true
I20260620 11:33:21.509087 5189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbfc0247cf764be3a74fd705b9e054c9" candidate_uuid: "c03da774dea24698856551d74863f322" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a246165e5246ec8680e4d4ef91dcf9" is_pre_election: true
I20260620 11:33:21.509150 4909 raft_consensus.cc:2454] T fbfc0247cf764be3a74fd705b9e054c9 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c03da774dea24698856551d74863f322 in term 0.
I20260620 11:33:21.509191 5189 raft_consensus.cc:2454] T fbfc0247cf764be3a74fd705b9e054c9 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c03da774dea24698856551d74863f322 in term 0.
I20260620 11:33:21.509291 4982 leader_election.cc:302] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15d99a934eee48eca6d992034c546031, c03da774dea24698856551d74863f322; no voters:
I20260620 11:33:21.509361 5280 raft_consensus.cc:2781] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:33:21.509389 5280 raft_consensus.cc:482] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:33:21.509410 5280 raft_consensus.cc:3037] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:21.510298 5280 raft_consensus.cc:504] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.510429 5280 leader_election.cc:288] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322 [CANDIDATE]: Term 1 election: Requested vote from peers 15d99a934eee48eca6d992034c546031 (127.2.243.129:44715), 00a246165e5246ec8680e4d4ef91dcf9 (127.2.243.131:46595)
I20260620 11:33:21.510582 5189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbfc0247cf764be3a74fd705b9e054c9" candidate_uuid: "c03da774dea24698856551d74863f322" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a246165e5246ec8680e4d4ef91dcf9"
I20260620 11:33:21.510598 4909 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fbfc0247cf764be3a74fd705b9e054c9" candidate_uuid: "c03da774dea24698856551d74863f322" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15d99a934eee48eca6d992034c546031"
I20260620 11:33:21.510658 5189 raft_consensus.cc:3037] T fbfc0247cf764be3a74fd705b9e054c9 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:21.510674 4909 raft_consensus.cc:3037] T fbfc0247cf764be3a74fd705b9e054c9 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:21.511443 4909 raft_consensus.cc:2454] T fbfc0247cf764be3a74fd705b9e054c9 P 15d99a934eee48eca6d992034c546031 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c03da774dea24698856551d74863f322 in term 1.
I20260620 11:33:21.511515 5189 raft_consensus.cc:2454] T fbfc0247cf764be3a74fd705b9e054c9 P 00a246165e5246ec8680e4d4ef91dcf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c03da774dea24698856551d74863f322 in term 1.
I20260620 11:33:21.511590 4982 leader_election.cc:302] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15d99a934eee48eca6d992034c546031, c03da774dea24698856551d74863f322; no voters:
I20260620 11:33:21.511658 5280 raft_consensus.cc:2781] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:21.511711 5280 raft_consensus.cc:686] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322 [term 1 LEADER]: Becoming Leader. State: Replica: c03da774dea24698856551d74863f322, State: Running, Role: LEADER
I20260620 11:33:21.511761 5280 consensus_queue.cc:237] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:21.512236 4750 catalog_manager.cc:5495] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322 reported cstate change: term changed from 0 to 1, leader changed from <none> to c03da774dea24698856551d74863f322 (127.2.243.130). New cstate: current_term: 1 leader_uuid: "c03da774dea24698856551d74863f322" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } health_report { overall_health: UNKNOWN } } }
I20260620 11:33:21.520821 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.517928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.2.243.130:42187)
0620 11:33:21.517965 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:21.517968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:21.517978 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:33:21.518001 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:21.518002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:21.518013 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:21.518078 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:21.518201 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.518982 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.520022 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.520362 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.520432 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.520575 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:21.520608 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:21.520660 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:21.520730 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 11:33:21.520751 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:33:21.525844 5256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.523083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:39533 (local address 127.2.243.130:42187)
0620 11:33:21.523114 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:21.523116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:21.523124 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:33:21.523164 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:21.523165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:21.523173 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:21.523225 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:21.523348 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.524012 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.525054 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.525307 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.525389 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.525527 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:21.525557 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:21.525608 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:21.525680 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 11:33:21.525718 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":16}
I20260620 11:33:21.525926 5101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:21.523027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:42187 (local address 127.2.243.130:39533)
0620 11:33:21.523070 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:21.523080 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:21.523113 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:21.523237 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:21.523238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:21.523239 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:33:21.523288 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.523290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.524103 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.524104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.524960 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:21.524963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:21.525389 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:21.525390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:21.525448 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:21.525453 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:33:21.525627 (+ 174us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:33:21.525629 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:33:21.525675 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 11:33:21.525746 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 11:33:21.528347 4909 raft_consensus.cc:1261] T fbfc0247cf764be3a74fd705b9e054c9 P 15d99a934eee48eca6d992034c546031 [term 1 FOLLOWER]: Refusing update from remote peer c03da774dea24698856551d74863f322: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:21.528352 5189 raft_consensus.cc:1261] T fbfc0247cf764be3a74fd705b9e054c9 P 00a246165e5246ec8680e4d4ef91dcf9 [term 1 FOLLOWER]: Refusing update from remote peer c03da774dea24698856551d74863f322: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:21.528548 5280 consensus_queue.cc:1035] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:21.528648 5280 consensus_queue.cc:1035] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 11:33:21.534256 5285 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 11:33:23.053941 4752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:23.054045 4752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37284:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:23.054826 4752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2: Sending DeleteTablet for 3 replicas of tablet fbfc0247cf764be3a74fd705b9e054c9
I20260620 11:33:23.055117 5010 tablet_service.cc:1514] Processing DeleteTablet for tablet fbfc0247cf764be3a74fd705b9e054c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:23 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:41014
I20260620 11:33:23.055158 4889 tablet_service.cc:1514] Processing DeleteTablet for tablet fbfc0247cf764be3a74fd705b9e054c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:23 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:38372
I20260620 11:33:23.055172 5169 tablet_service.cc:1514] Processing DeleteTablet for tablet fbfc0247cf764be3a74fd705b9e054c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:23 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:37588
I20260620 11:33:23.055363 5333 tablet_replica.cc:330] T fbfc0247cf764be3a74fd705b9e054c9 P 15d99a934eee48eca6d992034c546031: stopping tablet replica
I20260620 11:33:23.055370 5332 tablet_replica.cc:330] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322: stopping tablet replica
I20260620 11:33:23.055418 5333 raft_consensus.cc:2229] T fbfc0247cf764be3a74fd705b9e054c9 P 15d99a934eee48eca6d992034c546031 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:23.055418 5332 raft_consensus.cc:2229] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:33:23.055457 5333 raft_consensus.cc:2258] T fbfc0247cf764be3a74fd705b9e054c9 P 15d99a934eee48eca6d992034c546031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:23.055508 5332 raft_consensus.cc:2258] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:23.055753 5332 ts_tablet_manager.cc:1905] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:23.055769 5333 ts_tablet_manager.cc:1905] T fbfc0247cf764be3a74fd705b9e054c9 P 15d99a934eee48eca6d992034c546031: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:23.057433 5333 ts_tablet_manager.cc:1918] T fbfc0247cf764be3a74fd705b9e054c9 P 15d99a934eee48eca6d992034c546031: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 11:33:23.057479 5332 ts_tablet_manager.cc:1918] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 11:33:23.057503 5333 log.cc:1196] T fbfc0247cf764be3a74fd705b9e054c9 P 15d99a934eee48eca6d992034c546031: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-0/wal/wals/fbfc0247cf764be3a74fd705b9e054c9
I20260620 11:33:23.057529 5332 log.cc:1196] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-1/wal/wals/fbfc0247cf764be3a74fd705b9e054c9
I20260620 11:33:23.057791 5333 ts_tablet_manager.cc:1939] T fbfc0247cf764be3a74fd705b9e054c9 P 15d99a934eee48eca6d992034c546031: Deleting consensus metadata
I20260620 11:33:23.057791 5332 ts_tablet_manager.cc:1939] T fbfc0247cf764be3a74fd705b9e054c9 P c03da774dea24698856551d74863f322: Deleting consensus metadata
I20260620 11:33:23.057340 5334 tablet_replica.cc:330] T fbfc0247cf764be3a74fd705b9e054c9 P 00a246165e5246ec8680e4d4ef91dcf9: stopping tablet replica
I20260620 11:33:23.057973 5334 raft_consensus.cc:2229] T fbfc0247cf764be3a74fd705b9e054c9 P 00a246165e5246ec8680e4d4ef91dcf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:23.058017 5334 raft_consensus.cc:2258] T fbfc0247cf764be3a74fd705b9e054c9 P 00a246165e5246ec8680e4d4ef91dcf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:23.058137 4736 catalog_manager.cc:4841] TS c03da774dea24698856551d74863f322 (127.2.243.130:42187): tablet fbfc0247cf764be3a74fd705b9e054c9 (table test-table [id=cdfbb7a920eb4264831cf1543210bdcf]) successfully deleted
I20260620 11:33:23.059085 5334 ts_tablet_manager.cc:1905] T fbfc0247cf764be3a74fd705b9e054c9 P 00a246165e5246ec8680e4d4ef91dcf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:23.060329 4738 catalog_manager.cc:4841] TS 15d99a934eee48eca6d992034c546031 (127.2.243.129:44715): tablet fbfc0247cf764be3a74fd705b9e054c9 (table test-table [id=cdfbb7a920eb4264831cf1543210bdcf]) successfully deleted
I20260620 11:33:23.060716 5334 ts_tablet_manager.cc:1918] T fbfc0247cf764be3a74fd705b9e054c9 P 00a246165e5246ec8680e4d4ef91dcf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 11:33:23.060787 5334 log.cc:1196] T fbfc0247cf764be3a74fd705b9e054c9 P 00a246165e5246ec8680e4d4ef91dcf9: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/fbfc0247cf764be3a74fd705b9e054c9
I20260620 11:33:23.060998 5334 ts_tablet_manager.cc:1939] T fbfc0247cf764be3a74fd705b9e054c9 P 00a246165e5246ec8680e4d4ef91dcf9: Deleting consensus metadata
I20260620 11:33:23.061317 4737 catalog_manager.cc:4841] TS 00a246165e5246ec8680e4d4ef91dcf9 (127.2.243.131:46595): tablet fbfc0247cf764be3a74fd705b9e054c9 (table test-table [id=cdfbb7a920eb4264831cf1543210bdcf]) successfully deleted
Jun 20 11:33:23 dist-test-slave-ctbw krb5kdc[4707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955203, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 11:33:23 dist-test-slave-ctbw krb5kdc[4707](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955203, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:23.079507 5342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:23.072230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41624 (local address 127.2.243.190:34259)
0620 11:33:23.072378 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:23.072380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:23.072470 (+ 90us) server_negotiation.cc:407] Connection header received
0620 11:33:23.072573 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:23.072574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:23.072586 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:23.072657 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:23.073224 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:23.073736 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:23.074497 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:23.074709 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:23.074796 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:23.077078 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:23.077080 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:23.077080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:23.077087 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:23.078373 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:23.078676 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:23.078677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:23.078677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:23.078727 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:23.079068 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:23.079069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:23.079069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:23.079178 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:23.079254 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:23.079369 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 11:33:23.079437 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260620 11:33:23.080899 4752 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:33:23.081037 4752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:33:23.082273 5010 tablet_service.cc:1467] Processing CreateTablet for tablet a1622a3815f54c5fa7e7e81b8d182197 (DEFAULT_TABLE table=test-table [id=689724db6e6f4c95bf147836734d539f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:23.082291 5169 tablet_service.cc:1467] Processing CreateTablet for tablet a1622a3815f54c5fa7e7e81b8d182197 (DEFAULT_TABLE table=test-table [id=689724db6e6f4c95bf147836734d539f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:23.082293 4889 tablet_service.cc:1467] Processing CreateTablet for tablet a1622a3815f54c5fa7e7e81b8d182197 (DEFAULT_TABLE table=test-table [id=689724db6e6f4c95bf147836734d539f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:23.082427 4889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1622a3815f54c5fa7e7e81b8d182197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:23.082428 5010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1622a3815f54c5fa7e7e81b8d182197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:23.082474 5169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1622a3815f54c5fa7e7e81b8d182197. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:23.083818 5344 tablet_bootstrap.cc:492] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031: Bootstrap starting.
I20260620 11:33:23.083884 5343 tablet_bootstrap.cc:492] T a1622a3815f54c5fa7e7e81b8d182197 P c03da774dea24698856551d74863f322: Bootstrap starting.
I20260620 11:33:23.083951 5345 tablet_bootstrap.cc:492] T a1622a3815f54c5fa7e7e81b8d182197 P 00a246165e5246ec8680e4d4ef91dcf9: Bootstrap starting.
I20260620 11:33:23.084273 5343 tablet_bootstrap.cc:654] T a1622a3815f54c5fa7e7e81b8d182197 P c03da774dea24698856551d74863f322: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:23.084337 5345 tablet_bootstrap.cc:654] T a1622a3815f54c5fa7e7e81b8d182197 P 00a246165e5246ec8680e4d4ef91dcf9: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:23.084836 5344 tablet_bootstrap.cc:654] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:23.084849 5343 tablet_bootstrap.cc:492] T a1622a3815f54c5fa7e7e81b8d182197 P c03da774dea24698856551d74863f322: No bootstrap required, opened a new log
I20260620 11:33:23.084901 5343 ts_tablet_manager.cc:1397] T a1622a3815f54c5fa7e7e81b8d182197 P c03da774dea24698856551d74863f322: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:23.085055 5343 raft_consensus.cc:348] T a1622a3815f54c5fa7e7e81b8d182197 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:23.085127 5343 raft_consensus.cc:374] T a1622a3815f54c5fa7e7e81b8d182197 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:23.085141 5343 raft_consensus.cc:729] T a1622a3815f54c5fa7e7e81b8d182197 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03da774dea24698856551d74863f322, State: Initialized, Role: FOLLOWER
I20260620 11:33:23.085232 5343 consensus_queue.cc:260] T a1622a3815f54c5fa7e7e81b8d182197 P c03da774dea24698856551d74863f322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:23.085373 5343 ts_tablet_manager.cc:1428] T a1622a3815f54c5fa7e7e81b8d182197 P c03da774dea24698856551d74863f322: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:23.085481 5344 tablet_bootstrap.cc:492] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031: No bootstrap required, opened a new log
I20260620 11:33:23.085536 5344 ts_tablet_manager.cc:1397] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:33:23.085642 5344 raft_consensus.cc:348] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:23.085714 5344 raft_consensus.cc:374] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:23.085734 5344 raft_consensus.cc:729] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d99a934eee48eca6d992034c546031, State: Initialized, Role: FOLLOWER
I20260620 11:33:23.085776 5344 consensus_queue.cc:260] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:23.085947 5344 ts_tablet_manager.cc:1428] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:23.086161 5345 tablet_bootstrap.cc:492] T a1622a3815f54c5fa7e7e81b8d182197 P 00a246165e5246ec8680e4d4ef91dcf9: No bootstrap required, opened a new log
I20260620 11:33:23.086218 5345 ts_tablet_manager.cc:1397] T a1622a3815f54c5fa7e7e81b8d182197 P 00a246165e5246ec8680e4d4ef91dcf9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:33:23.086326 5345 raft_consensus.cc:348] T a1622a3815f54c5fa7e7e81b8d182197 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:23.086397 5345 raft_consensus.cc:374] T a1622a3815f54c5fa7e7e81b8d182197 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:23.086424 5345 raft_consensus.cc:729] T a1622a3815f54c5fa7e7e81b8d182197 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a246165e5246ec8680e4d4ef91dcf9, State: Initialized, Role: FOLLOWER
I20260620 11:33:23.086476 5345 consensus_queue.cc:260] T a1622a3815f54c5fa7e7e81b8d182197 P 00a246165e5246ec8680e4d4ef91dcf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:23.086668 5345 ts_tablet_manager.cc:1428] T a1622a3815f54c5fa7e7e81b8d182197 P 00a246165e5246ec8680e4d4ef91dcf9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:23.143139 5349 raft_consensus.cc:482] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:33:23.143239 5349 raft_consensus.cc:504] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:23.143599 5349 leader_election.cc:288] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c03da774dea24698856551d74863f322 (127.2.243.130:42187), 00a246165e5246ec8680e4d4ef91dcf9 (127.2.243.131:46595)
I20260620 11:33:23.147058 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:23.143829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:58403 (local address 127.2.243.131:46595)
0620 11:33:23.144020 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:23.144022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:23.144054 (+ 32us) server_negotiation.cc:407] Connection header received
0620 11:33:23.144141 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:23.144143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:23.144152 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:23.144239 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:23.144425 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:23.144909 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:23.146320 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:23.146694 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:23.146780 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:23.146789 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:23.146955 (+ 166us) server_negotiation.cc:299] Negotiation successful
0620 11:33:23.146992 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":116,"threads_started":1}
I20260620 11:33:23.147068 5351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:23.143778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:46595 (local address 127.2.243.129:58403)
0620 11:33:23.143984 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:23.143998 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:23.144066 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:23.144243 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:23.144244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:23.144246 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:23.144324 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:23.144325 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:23.145002 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:23.145003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:23.146229 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:23.146231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:23.146777 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:23.146778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:23.146852 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:23.146856 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:33:23.146937 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 11:33:23.146976 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":20,"thread_start_us":119,"threads_started":1}
I20260620 11:33:23.147217 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:23.143787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:42187 (local address 127.2.243.129:58797)
0620 11:33:23.144059 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:23.144070 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:23.144165 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:23.144408 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:23.144409 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:23.144411 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:23.144493 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:23.144495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:23.145396 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:23.145397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:23.146289 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:23.146291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:23.146851 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:23.146852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:23.146980 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:23.146983 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:33:23.147063 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 11:33:23.147100 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":16,"thread_start_us":193,"threads_started":1}
I20260620 11:33:23.147295 5188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1622a3815f54c5fa7e7e81b8d182197" candidate_uuid: "15d99a934eee48eca6d992034c546031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a246165e5246ec8680e4d4ef91dcf9" is_pre_election: true
I20260620 11:33:23.147377 5188 raft_consensus.cc:2454] T a1622a3815f54c5fa7e7e81b8d182197 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15d99a934eee48eca6d992034c546031 in term 0.
I20260620 11:33:23.147544 4841 leader_election.cc:302] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a246165e5246ec8680e4d4ef91dcf9, 15d99a934eee48eca6d992034c546031; no voters:
I20260620 11:33:23.147598 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:23.143881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:58797 (local address 127.2.243.130:42187)
0620 11:33:23.144048 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:23.144051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:23.144147 (+ 96us) server_negotiation.cc:407] Connection header received
0620 11:33:23.144268 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:23.144269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:23.144289 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:23.144403 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:23.144578 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:23.145293 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:23.146383 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:23.146767 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:23.146852 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:23.146860 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:23.147072 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 11:33:23.147520 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
I20260620 11:33:23.147665 5349 raft_consensus.cc:2781] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:33:23.147701 5349 raft_consensus.cc:482] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:33:23.147739 5349 raft_consensus.cc:3037] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:23.147789 5049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1622a3815f54c5fa7e7e81b8d182197" candidate_uuid: "15d99a934eee48eca6d992034c546031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03da774dea24698856551d74863f322" is_pre_election: true
I20260620 11:33:23.147884 5049 raft_consensus.cc:2454] T a1622a3815f54c5fa7e7e81b8d182197 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15d99a934eee48eca6d992034c546031 in term 0.
I20260620 11:33:23.148468 5349 raft_consensus.cc:504] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:23.148622 5349 leader_election.cc:288] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031 [CANDIDATE]: Term 1 election: Requested vote from peers c03da774dea24698856551d74863f322 (127.2.243.130:42187), 00a246165e5246ec8680e4d4ef91dcf9 (127.2.243.131:46595)
I20260620 11:33:23.148752 5188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1622a3815f54c5fa7e7e81b8d182197" candidate_uuid: "15d99a934eee48eca6d992034c546031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a246165e5246ec8680e4d4ef91dcf9"
I20260620 11:33:23.148821 5188 raft_consensus.cc:3037] T a1622a3815f54c5fa7e7e81b8d182197 P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:23.148765 5049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1622a3815f54c5fa7e7e81b8d182197" candidate_uuid: "15d99a934eee48eca6d992034c546031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03da774dea24698856551d74863f322"
I20260620 11:33:23.148890 5049 raft_consensus.cc:3037] T a1622a3815f54c5fa7e7e81b8d182197 P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:23.149443 5188 raft_consensus.cc:2454] T a1622a3815f54c5fa7e7e81b8d182197 P 00a246165e5246ec8680e4d4ef91dcf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15d99a934eee48eca6d992034c546031 in term 1.
I20260620 11:33:23.149591 5049 raft_consensus.cc:2454] T a1622a3815f54c5fa7e7e81b8d182197 P c03da774dea24698856551d74863f322 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15d99a934eee48eca6d992034c546031 in term 1.
I20260620 11:33:23.149592 4841 leader_election.cc:302] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a246165e5246ec8680e4d4ef91dcf9, 15d99a934eee48eca6d992034c546031; no voters:
I20260620 11:33:23.149722 5349 raft_consensus.cc:2781] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:23.149881 5349 raft_consensus.cc:686] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031 [term 1 LEADER]: Becoming Leader. State: Replica: 15d99a934eee48eca6d992034c546031, State: Running, Role: LEADER
I20260620 11:33:23.149941 5349 consensus_queue.cc:237] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:23.150679 4750 catalog_manager.cc:5495] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15d99a934eee48eca6d992034c546031 (127.2.243.129). New cstate: current_term: 1 leader_uuid: "15d99a934eee48eca6d992034c546031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } health_report { overall_health: UNKNOWN } } }
I20260620 11:33:23.163130 5358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:23.160634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50674 (local address 127.2.243.129:44715)
0620 11:33:23.160788 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:23.160790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:23.160798 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:33:23.160875 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:23.160876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:23.160896 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:23.160993 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:23.161112 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:23.161612 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:23.162307 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:23.162517 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:23.162570 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:23.162711 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:23.162816 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:23.162925 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:23.162999 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 11:33:23.163015 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260620 11:33:23.168234 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:23.165929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:58879 (local address 127.2.243.130:42187)
0620 11:33:23.165989 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:23.165991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:23.166042 (+ 51us) server_negotiation.cc:407] Connection header received
0620 11:33:23.166098 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:23.166098 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:23.166106 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:23.166151 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:23.166235 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:23.166679 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:23.167521 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:23.167721 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:23.167807 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:23.167969 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:23.168007 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:23.168068 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:23.168133 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 11:33:23.168155 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 11:33:23.168248 5361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:23.165770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:42187 (local address 127.2.243.129:58879)
0620 11:33:23.165959 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:23.165971 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:23.166057 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:23.166154 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:23.166154 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:23.166155 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:33:23.166201 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:23.166202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:23.166745 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:23.166746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:23.167424 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:23.167426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:23.167806 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:23.167807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:23.167872 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:23.167885 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:33:23.168069 (+ 184us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:33:23.168071 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:33:23.168121 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 11:33:23.168161 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":84,"threads_started":1}
I20260620 11:33:23.170368 5049 raft_consensus.cc:1261] T a1622a3815f54c5fa7e7e81b8d182197 P c03da774dea24698856551d74863f322 [term 1 FOLLOWER]: Refusing update from remote peer 15d99a934eee48eca6d992034c546031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:23.170450 5188 raft_consensus.cc:1261] T a1622a3815f54c5fa7e7e81b8d182197 P 00a246165e5246ec8680e4d4ef91dcf9 [term 1 FOLLOWER]: Refusing update from remote peer 15d99a934eee48eca6d992034c546031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:23.170652 5355 consensus_queue.cc:1035] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:23.170769 5349 consensus_queue.cc:1035] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:23.177942 5358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:23.175538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:43373 (local address 127.2.243.129:44715)
0620 11:33:23.175583 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:23.175586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:23.175696 (+ 110us) server_negotiation.cc:407] Connection header received
0620 11:33:23.175769 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:23.175769 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:23.175779 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:23.175819 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:23.175948 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:23.176419 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:23.177151 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:23.177455 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:23.177524 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:23.177708 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:23.177741 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:23.177782 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:23.177850 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 11:33:23.177871 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:33:23.177973 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:23.175516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:44715 (local address 127.2.243.130:43373)
0620 11:33:23.175643 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:23.175655 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:23.175712 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:23.175830 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:23.175830 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:23.175832 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:33:23.175878 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:23.175879 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:23.176504 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:23.176505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:23.177069 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:23.177070 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:23.177566 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:23.177566 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:23.177623 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:23.177628 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:33:23.177793 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:33:23.177794 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 11:33:23.177846 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 11:33:23.177887 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":82,"threads_started":1}
I20260620 11:33:23.184527 4750 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:23.184659 4750 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:23.185331 4750 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2: Sending DeleteTablet for 3 replicas of tablet a1622a3815f54c5fa7e7e81b8d182197
I20260620 11:33:23.185617 4889 tablet_service.cc:1514] Processing DeleteTablet for tablet a1622a3815f54c5fa7e7e81b8d182197 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:23 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:38372
I20260620 11:33:23.185671 5169 tablet_service.cc:1514] Processing DeleteTablet for tablet a1622a3815f54c5fa7e7e81b8d182197 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:23 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:37588
I20260620 11:33:23.185739 5333 tablet_replica.cc:330] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031: stopping tablet replica
I20260620 11:33:23.185760 5334 tablet_replica.cc:330] T a1622a3815f54c5fa7e7e81b8d182197 P 00a246165e5246ec8680e4d4ef91dcf9: stopping tablet replica
I20260620 11:33:23.185801 5333 raft_consensus.cc:2229] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:33:23.185819 5334 raft_consensus.cc:2229] T a1622a3815f54c5fa7e7e81b8d182197 P 00a246165e5246ec8680e4d4ef91dcf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:23.185909 5333 raft_consensus.cc:2258] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:23.185940 5334 raft_consensus.cc:2258] T a1622a3815f54c5fa7e7e81b8d182197 P 00a246165e5246ec8680e4d4ef91dcf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:23.186247 5010 tablet_service.cc:1514] Processing DeleteTablet for tablet a1622a3815f54c5fa7e7e81b8d182197 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:23 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:41014
I20260620 11:33:23.186266 5334 ts_tablet_manager.cc:1905] T a1622a3815f54c5fa7e7e81b8d182197 P 00a246165e5246ec8680e4d4ef91dcf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:23.186367 5332 tablet_replica.cc:330] T a1622a3815f54c5fa7e7e81b8d182197 P c03da774dea24698856551d74863f322: stopping tablet replica
I20260620 11:33:23.186435 5332 raft_consensus.cc:2229] T a1622a3815f54c5fa7e7e81b8d182197 P c03da774dea24698856551d74863f322 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:23.186478 5332 raft_consensus.cc:2258] T a1622a3815f54c5fa7e7e81b8d182197 P c03da774dea24698856551d74863f322 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:23.186734 5333 ts_tablet_manager.cc:1905] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:23.188059 5334 ts_tablet_manager.cc:1918] T a1622a3815f54c5fa7e7e81b8d182197 P 00a246165e5246ec8680e4d4ef91dcf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:33:23.188129 5334 log.cc:1196] T a1622a3815f54c5fa7e7e81b8d182197 P 00a246165e5246ec8680e4d4ef91dcf9: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/a1622a3815f54c5fa7e7e81b8d182197
I20260620 11:33:23.188371 5334 ts_tablet_manager.cc:1939] T a1622a3815f54c5fa7e7e81b8d182197 P 00a246165e5246ec8680e4d4ef91dcf9: Deleting consensus metadata
I20260620 11:33:23.188408 5333 ts_tablet_manager.cc:1918] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:33:23.188480 5333 log.cc:1196] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-0/wal/wals/a1622a3815f54c5fa7e7e81b8d182197
I20260620 11:33:23.188709 5333 ts_tablet_manager.cc:1939] T a1622a3815f54c5fa7e7e81b8d182197 P 15d99a934eee48eca6d992034c546031: Deleting consensus metadata
I20260620 11:33:23.188711 4737 catalog_manager.cc:4841] TS 00a246165e5246ec8680e4d4ef91dcf9 (127.2.243.131:46595): tablet a1622a3815f54c5fa7e7e81b8d182197 (table test-table [id=689724db6e6f4c95bf147836734d539f]) successfully deleted
I20260620 11:33:23.189021 4738 catalog_manager.cc:4841] TS 15d99a934eee48eca6d992034c546031 (127.2.243.129:44715): tablet a1622a3815f54c5fa7e7e81b8d182197 (table test-table [id=689724db6e6f4c95bf147836734d539f]) successfully deleted
I20260620 11:33:23.189057 5332 ts_tablet_manager.cc:1905] T a1622a3815f54c5fa7e7e81b8d182197 P c03da774dea24698856551d74863f322: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:23.190182 5332 ts_tablet_manager.cc:1918] T a1622a3815f54c5fa7e7e81b8d182197 P c03da774dea24698856551d74863f322: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:33:23.190253 5332 log.cc:1196] T a1622a3815f54c5fa7e7e81b8d182197 P c03da774dea24698856551d74863f322: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-1/wal/wals/a1622a3815f54c5fa7e7e81b8d182197
I20260620 11:33:23.190493 5332 ts_tablet_manager.cc:1939] T a1622a3815f54c5fa7e7e81b8d182197 P c03da774dea24698856551d74863f322: Deleting consensus metadata
I20260620 11:33:23.190532 4750 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:33:23.190699 4750 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:33:23.190820 4736 catalog_manager.cc:4841] TS c03da774dea24698856551d74863f322 (127.2.243.130:42187): tablet a1622a3815f54c5fa7e7e81b8d182197 (table test-table [id=689724db6e6f4c95bf147836734d539f]) successfully deleted
I20260620 11:33:23.192489 5010 tablet_service.cc:1467] Processing CreateTablet for tablet 0b62eff8cd5c4e65b20a9b8a22cabbac (DEFAULT_TABLE table=test-table [id=150ff2bdee8d48aea16a08451ae4b2bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:23.192512 5169 tablet_service.cc:1467] Processing CreateTablet for tablet 0b62eff8cd5c4e65b20a9b8a22cabbac (DEFAULT_TABLE table=test-table [id=150ff2bdee8d48aea16a08451ae4b2bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:23.192626 5169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b62eff8cd5c4e65b20a9b8a22cabbac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:23.192626 5010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b62eff8cd5c4e65b20a9b8a22cabbac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:23.193883 5343 tablet_bootstrap.cc:492] T 0b62eff8cd5c4e65b20a9b8a22cabbac P c03da774dea24698856551d74863f322: Bootstrap starting.
I20260620 11:33:23.193964 4889 tablet_service.cc:1467] Processing CreateTablet for tablet 0b62eff8cd5c4e65b20a9b8a22cabbac (DEFAULT_TABLE table=test-table [id=150ff2bdee8d48aea16a08451ae4b2bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:23.194089 4889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b62eff8cd5c4e65b20a9b8a22cabbac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:23.194221 5343 tablet_bootstrap.cc:654] T 0b62eff8cd5c4e65b20a9b8a22cabbac P c03da774dea24698856551d74863f322: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:23.194439 5345 tablet_bootstrap.cc:492] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 00a246165e5246ec8680e4d4ef91dcf9: Bootstrap starting.
I20260620 11:33:23.194752 5345 tablet_bootstrap.cc:654] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 00a246165e5246ec8680e4d4ef91dcf9: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:23.194751 5343 tablet_bootstrap.cc:492] T 0b62eff8cd5c4e65b20a9b8a22cabbac P c03da774dea24698856551d74863f322: No bootstrap required, opened a new log
I20260620 11:33:23.194798 5343 ts_tablet_manager.cc:1397] T 0b62eff8cd5c4e65b20a9b8a22cabbac P c03da774dea24698856551d74863f322: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:23.194947 5343 raft_consensus.cc:348] T 0b62eff8cd5c4e65b20a9b8a22cabbac P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:23.195032 5343 raft_consensus.cc:374] T 0b62eff8cd5c4e65b20a9b8a22cabbac P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:23.195060 5343 raft_consensus.cc:729] T 0b62eff8cd5c4e65b20a9b8a22cabbac P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c03da774dea24698856551d74863f322, State: Initialized, Role: FOLLOWER
I20260620 11:33:23.195109 5343 consensus_queue.cc:260] T 0b62eff8cd5c4e65b20a9b8a22cabbac P c03da774dea24698856551d74863f322 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:23.195235 5343 ts_tablet_manager.cc:1428] T 0b62eff8cd5c4e65b20a9b8a22cabbac P c03da774dea24698856551d74863f322: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:23.195355 5344 tablet_bootstrap.cc:492] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031: Bootstrap starting.
I20260620 11:33:23.195761 5344 tablet_bootstrap.cc:654] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:23.195815 5345 tablet_bootstrap.cc:492] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 00a246165e5246ec8680e4d4ef91dcf9: No bootstrap required, opened a new log
I20260620 11:33:23.195858 5345 ts_tablet_manager.cc:1397] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 00a246165e5246ec8680e4d4ef91dcf9: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:23.195961 5345 raft_consensus.cc:348] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:23.196031 5345 raft_consensus.cc:374] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:23.196054 5345 raft_consensus.cc:729] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00a246165e5246ec8680e4d4ef91dcf9, State: Initialized, Role: FOLLOWER
I20260620 11:33:23.196094 5345 consensus_queue.cc:260] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 00a246165e5246ec8680e4d4ef91dcf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:23.196192 5345 ts_tablet_manager.cc:1428] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 00a246165e5246ec8680e4d4ef91dcf9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:23.196784 5344 tablet_bootstrap.cc:492] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031: No bootstrap required, opened a new log
I20260620 11:33:23.196853 5344 ts_tablet_manager.cc:1397] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:33:23.197000 5344 raft_consensus.cc:348] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:23.197073 5344 raft_consensus.cc:374] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:23.197101 5344 raft_consensus.cc:729] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15d99a934eee48eca6d992034c546031, State: Initialized, Role: FOLLOWER
I20260620 11:33:23.197157 5344 consensus_queue.cc:260] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:23.197280 5344 ts_tablet_manager.cc:1428] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:23.420779 5364 raft_consensus.cc:482] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:33:23.420893 5364 raft_consensus.cc:504] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:23.421120 5364 leader_election.cc:288] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c03da774dea24698856551d74863f322 (127.2.243.130:42187), 00a246165e5246ec8680e4d4ef91dcf9 (127.2.243.131:46595)
I20260620 11:33:23.421330 5188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b62eff8cd5c4e65b20a9b8a22cabbac" candidate_uuid: "15d99a934eee48eca6d992034c546031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a246165e5246ec8680e4d4ef91dcf9" is_pre_election: true
I20260620 11:33:23.421317 5049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b62eff8cd5c4e65b20a9b8a22cabbac" candidate_uuid: "15d99a934eee48eca6d992034c546031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03da774dea24698856551d74863f322" is_pre_election: true
I20260620 11:33:23.421443 5188 raft_consensus.cc:2454] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15d99a934eee48eca6d992034c546031 in term 0.
I20260620 11:33:23.421442 5049 raft_consensus.cc:2454] T 0b62eff8cd5c4e65b20a9b8a22cabbac P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 15d99a934eee48eca6d992034c546031 in term 0.
I20260620 11:33:23.421636 4841 leader_election.cc:302] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00a246165e5246ec8680e4d4ef91dcf9, 15d99a934eee48eca6d992034c546031; no voters:
I20260620 11:33:23.421731 5364 raft_consensus.cc:2781] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:33:23.421775 5364 raft_consensus.cc:482] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:33:23.421794 5364 raft_consensus.cc:3037] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:23.422586 5364 raft_consensus.cc:504] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:23.422720 5364 leader_election.cc:288] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031 [CANDIDATE]: Term 1 election: Requested vote from peers c03da774dea24698856551d74863f322 (127.2.243.130:42187), 00a246165e5246ec8680e4d4ef91dcf9 (127.2.243.131:46595)
I20260620 11:33:23.422849 5049 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b62eff8cd5c4e65b20a9b8a22cabbac" candidate_uuid: "15d99a934eee48eca6d992034c546031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c03da774dea24698856551d74863f322"
I20260620 11:33:23.422951 5049 raft_consensus.cc:3037] T 0b62eff8cd5c4e65b20a9b8a22cabbac P c03da774dea24698856551d74863f322 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:23.422919 5188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b62eff8cd5c4e65b20a9b8a22cabbac" candidate_uuid: "15d99a934eee48eca6d992034c546031" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00a246165e5246ec8680e4d4ef91dcf9"
I20260620 11:33:23.423014 5188 raft_consensus.cc:3037] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 00a246165e5246ec8680e4d4ef91dcf9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:23.423545 5049 raft_consensus.cc:2454] T 0b62eff8cd5c4e65b20a9b8a22cabbac P c03da774dea24698856551d74863f322 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15d99a934eee48eca6d992034c546031 in term 1.
I20260620 11:33:23.423677 5188 raft_consensus.cc:2454] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 00a246165e5246ec8680e4d4ef91dcf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 15d99a934eee48eca6d992034c546031 in term 1.
I20260620 11:33:23.423761 4840 leader_election.cc:302] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15d99a934eee48eca6d992034c546031, c03da774dea24698856551d74863f322; no voters:
I20260620 11:33:23.423848 5364 raft_consensus.cc:2781] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:23.423907 5364 raft_consensus.cc:686] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031 [term 1 LEADER]: Becoming Leader. State: Replica: 15d99a934eee48eca6d992034c546031, State: Running, Role: LEADER
I20260620 11:33:23.423960 5364 consensus_queue.cc:237] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } }
I20260620 11:33:23.424590 4750 catalog_manager.cc:5495] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031 reported cstate change: term changed from 0 to 1, leader changed from <none> to 15d99a934eee48eca6d992034c546031 (127.2.243.129). New cstate: current_term: 1 leader_uuid: "15d99a934eee48eca6d992034c546031" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15d99a934eee48eca6d992034c546031" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44715 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 } health_report { overall_health: UNKNOWN } } }
I20260620 11:33:23.483237 5188 raft_consensus.cc:1261] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 00a246165e5246ec8680e4d4ef91dcf9 [term 1 FOLLOWER]: Refusing update from remote peer 15d99a934eee48eca6d992034c546031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:23.483261 5049 raft_consensus.cc:1261] T 0b62eff8cd5c4e65b20a9b8a22cabbac P c03da774dea24698856551d74863f322 [term 1 FOLLOWER]: Refusing update from remote peer 15d99a934eee48eca6d992034c546031: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:23.483532 5364 consensus_queue.cc:1035] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c03da774dea24698856551d74863f322" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:23.483656 5349 consensus_queue.cc:1035] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 15d99a934eee48eca6d992034c546031 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00a246165e5246ec8680e4d4ef91dcf9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 11:33:23.487910 5316 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 11:33:25.008018 4750 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:25.008142 4750 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:25.009042 4750 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40ffddc373b64bfba32915ea932127b2: Sending DeleteTablet for 3 replicas of tablet 0b62eff8cd5c4e65b20a9b8a22cabbac
I20260620 11:33:25.009342 5010 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b62eff8cd5c4e65b20a9b8a22cabbac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:25 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:41014
I20260620 11:33:25.009419 4889 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b62eff8cd5c4e65b20a9b8a22cabbac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:25 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:38372
I20260620 11:33:25.009426 5169 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b62eff8cd5c4e65b20a9b8a22cabbac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:25 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:37588
I20260620 11:33:25.009543 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 4824
I20260620 11:33:25.009615 5396 tablet_replica.cc:330] T 0b62eff8cd5c4e65b20a9b8a22cabbac P c03da774dea24698856551d74863f322: stopping tablet replica
I20260620 11:33:25.010228 5396 raft_consensus.cc:2229] T 0b62eff8cd5c4e65b20a9b8a22cabbac P c03da774dea24698856551d74863f322 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:25.010289 5396 raft_consensus.cc:2258] T 0b62eff8cd5c4e65b20a9b8a22cabbac P c03da774dea24698856551d74863f322 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:25.010536 5396 ts_tablet_manager.cc:1905] T 0b62eff8cd5c4e65b20a9b8a22cabbac P c03da774dea24698856551d74863f322: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:25.011827 5398 tablet_replica.cc:330] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 00a246165e5246ec8680e4d4ef91dcf9: stopping tablet replica
I20260620 11:33:25.011905 5398 raft_consensus.cc:2229] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 00a246165e5246ec8680e4d4ef91dcf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:25.011945 5398 raft_consensus.cc:2258] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 00a246165e5246ec8680e4d4ef91dcf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:25.012140 5398 ts_tablet_manager.cc:1905] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 00a246165e5246ec8680e4d4ef91dcf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:25.012523 5396 ts_tablet_manager.cc:1918] T 0b62eff8cd5c4e65b20a9b8a22cabbac P c03da774dea24698856551d74863f322: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 11:33:25.012588 5396 log.cc:1196] T 0b62eff8cd5c4e65b20a9b8a22cabbac P c03da774dea24698856551d74863f322: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-1/wal/wals/0b62eff8cd5c4e65b20a9b8a22cabbac
I20260620 11:33:25.012818 5396 ts_tablet_manager.cc:1939] T 0b62eff8cd5c4e65b20a9b8a22cabbac P c03da774dea24698856551d74863f322: Deleting consensus metadata
I20260620 11:33:25.013173 4736 catalog_manager.cc:4841] TS c03da774dea24698856551d74863f322 (127.2.243.130:42187): tablet 0b62eff8cd5c4e65b20a9b8a22cabbac (table test-table [id=150ff2bdee8d48aea16a08451ae4b2bb]) successfully deleted
I20260620 11:33:25.013607 5398 ts_tablet_manager.cc:1918] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 00a246165e5246ec8680e4d4ef91dcf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 11:33:25.013677 5398 log.cc:1196] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 00a246165e5246ec8680e4d4ef91dcf9: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/0b62eff8cd5c4e65b20a9b8a22cabbac
I20260620 11:33:25.013877 5398 ts_tablet_manager.cc:1939] T 0b62eff8cd5c4e65b20a9b8a22cabbac P 00a246165e5246ec8680e4d4ef91dcf9: Deleting consensus metadata
I20260620 11:33:25.014205 4737 catalog_manager.cc:4841] TS 00a246165e5246ec8680e4d4ef91dcf9 (127.2.243.131:46595): tablet 0b62eff8cd5c4e65b20a9b8a22cabbac (table test-table [id=150ff2bdee8d48aea16a08451ae4b2bb]) successfully deleted
W20260620 11:33:25.016666 4738 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.243.129:44715 (error 108)
I20260620 11:33:25.016955 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 4964
I20260620 11:33:25.017748 5400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:25.017025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:44715 (local address 127.0.0.1:50688)
0620 11:33:25.017652 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:25.017701 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.129:44715: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":59,"threads_started":1}
W20260620 11:33:25.017843 4738 catalog_manager.cc:4584] TS 15d99a934eee48eca6d992034c546031 (127.2.243.129:44715): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0b62eff8cd5c4e65b20a9b8a22cabbac: Network error: Client connection negotiation failed: client connection to 127.2.243.129:44715: connect: Connection refused (error 111)
I20260620 11:33:25.023442 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 5104
I20260620 11:33:25.028550 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 4720
2026-06-20T11:33:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5618 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:33:25 dist-test-slave-ctbw krb5kdc[5403](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:33:25 dist-test-slave-ctbw krb5kdc[5403](info): set up 2 sockets
Jun 20 11:33:25 dist-test-slave-ctbw krb5kdc[5403](info): commencing operation
krb5kdc: starting...
W20260620 11:33:26.098893 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:33:26 dist-test-slave-ctbw krb5kdc[5403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:33:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:33:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:33:26.215219 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:43275
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:42535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:43275
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:26.295171 5416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:26.295419 5416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:26.295455 5416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:26.296607 5416 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:33:26.296672 5416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:26.296700 5416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:26.296729 5416 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:33:26.296753 5416 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:33:26.298173 5416 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:42535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:43275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:43275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:26.298442 5416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:26.298671 5416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:26.303280 5422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:26.303356 5421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:26.303269 5424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:26.303480 5416 server_base.cc:1034] running on GCE node
I20260620 11:33:26.303759 5416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:26.304034 5416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:26.305191 5416 hybrid_clock.cc:648] HybridClock initialized: now 1781955206305180 us; error 41 us; skew 500 ppm
Jun 20 11:33:26 dist-test-slave-ctbw krb5kdc[5403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955206, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:26.307765 5416 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:33:26.308173 5416 webserver.cc:460] Webserver started at http://127.2.243.190:33787/ using document root <none> and password file <none>
I20260620 11:33:26.308394 5416 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:26.308465 5416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:26.308605 5416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:26.309686 5416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "846f84944ec14025a60dab4190ab9cd6"
format_stamp: "Formatted at 2026-06-20 11:33:26 on dist-test-slave-ctbw"
server_key: "827b17771ed361acd81fb09f9f3c6e56"
server_key_iv: "4ee8ecfa37a9c3c4dbc72fd53644bc1c"
server_key_version: "encryptionkey@0"
I20260620 11:33:26.310046 5416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "846f84944ec14025a60dab4190ab9cd6"
format_stamp: "Formatted at 2026-06-20 11:33:26 on dist-test-slave-ctbw"
server_key: "827b17771ed361acd81fb09f9f3c6e56"
server_key_iv: "4ee8ecfa37a9c3c4dbc72fd53644bc1c"
server_key_version: "encryptionkey@0"
I20260620 11:33:26.311501 5416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:33:26.312436 5431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:26.312645 5416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:26.312753 5416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "846f84944ec14025a60dab4190ab9cd6"
format_stamp: "Formatted at 2026-06-20 11:33:26 on dist-test-slave-ctbw"
server_key: "827b17771ed361acd81fb09f9f3c6e56"
server_key_iv: "4ee8ecfa37a9c3c4dbc72fd53644bc1c"
server_key_version: "encryptionkey@0"
I20260620 11:33:26.312834 5416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:26.331168 5416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:26.333524 5416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:26.333681 5416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:26.337630 5416 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:43275
I20260620 11:33:26.337669 5483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:43275 every 8 connection(s)
I20260620 11:33:26.338032 5416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:33:26.339028 5484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:26.339500 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 5416
I20260620 11:33:26.339596 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:33:26.339756 3022 external_mini_cluster.cc:1428] Setting key a8513d5d34f94b86f2359ab5b516447c
I20260620 11:33:26.342180 5484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6: Bootstrap starting.
I20260620 11:33:26.342818 5484 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:26.343148 5484 log.cc:824] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:26.343863 5484 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6: No bootstrap required, opened a new log
I20260620 11:33:26.345938 5484 raft_consensus.cc:348] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846f84944ec14025a60dab4190ab9cd6" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43275 } }
I20260620 11:33:26.346123 5484 raft_consensus.cc:374] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:26.346160 5484 raft_consensus.cc:729] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846f84944ec14025a60dab4190ab9cd6, State: Initialized, Role: FOLLOWER
Jun 20 11:33:26 dist-test-slave-ctbw krb5kdc[5403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955206, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:26.346300 5484 consensus_queue.cc:260] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846f84944ec14025a60dab4190ab9cd6" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43275 } }
I20260620 11:33:26.346401 5484 raft_consensus.cc:388] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:33:26.346436 5484 raft_consensus.cc:482] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:33:26.346486 5484 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:26.347211 5484 raft_consensus.cc:504] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846f84944ec14025a60dab4190ab9cd6" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43275 } }
I20260620 11:33:26.347340 5484 leader_election.cc:302] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 846f84944ec14025a60dab4190ab9cd6; no voters:
I20260620 11:33:26.347643 5484 leader_election.cc:288] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:33:26.347680 5489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:26.347862 5489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6 [term 1 LEADER]: Becoming Leader. State: Replica: 846f84944ec14025a60dab4190ab9cd6, State: Running, Role: LEADER
I20260620 11:33:26.347947 5489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846f84944ec14025a60dab4190ab9cd6" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43275 } }
I20260620 11:33:26.347967 5484 sys_catalog.cc:564] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:26.348264 5489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 846f84944ec14025a60dab4190ab9cd6. Latest consensus state: current_term: 1 leader_uuid: "846f84944ec14025a60dab4190ab9cd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846f84944ec14025a60dab4190ab9cd6" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43275 } } }
I20260620 11:33:26.348296 5490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "846f84944ec14025a60dab4190ab9cd6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846f84944ec14025a60dab4190ab9cd6" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43275 } } }
I20260620 11:33:26.348543 5490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6 [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:26.348435 5489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6 [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:26.348982 5498 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:33:26.349535 5498 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:33:26.351198 5498 catalog_manager.cc:1347] Generated new cluster ID: 2bc9da7083c742889d21c376738f4c91
I20260620 11:33:26.351244 5498 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:33:26.351817 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:26.340311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52442 (local address 127.2.243.190:43275)
0620 11:33:26.340469 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:26.340477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:26.340500 (+ 23us) server_negotiation.cc:407] Connection header received
0620 11:33:26.340889 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:26.340896 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:26.340941 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:26.341058 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:26.341931 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:26.342829 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:26.343725 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:26.344172 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:26.344282 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:26.346897 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:26.346901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:26.346901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:26.346914 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:26.350058 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:26.350496 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:26.350499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:26.350499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:26.350599 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:26.350838 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:26.350840 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:26.350840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:26.351160 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:26.351323 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:26.351447 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 11:33:26.351594 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
I20260620 11:33:26.375765 5498 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:33:26.376310 5498 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:33:26.382771 5498 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 846f84944ec14025a60dab4190ab9cd6: Generated new TSK 0
I20260620 11:33:26.382962 5498 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
I20260620 11:33:26.418135 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:43275
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:42535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:26.503834 5510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:26.504024 5510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:26.504046 5510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:26.505158 5510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:26.505199 5510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:26.505225 5510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:33:26.506603 5510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:42535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:43275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:26.506829 5510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:26.507071 5510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:26.511925 5515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:26.511970 5516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:26.512113 5510 server_base.cc:1034] running on GCE node
W20260620 11:33:26.511970 5518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:26.512470 5510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:26.512734 5510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:26.513900 5510 hybrid_clock.cc:648] HybridClock initialized: now 1781955206513879 us; error 35 us; skew 500 ppm
Jun 20 11:33:26 dist-test-slave-ctbw krb5kdc[5403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955206, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:26.516475 5510 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:33:26.516939 5510 webserver.cc:460] Webserver started at http://127.2.243.129:33141/ using document root <none> and password file <none>
I20260620 11:33:26.517118 5510 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:26.517172 5510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:26.517263 5510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:26.518211 5510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "2995a00c9f944adaa4c37d0cfa29d3aa"
format_stamp: "Formatted at 2026-06-20 11:33:26 on dist-test-slave-ctbw"
server_key: "4ceab65a6fa00c75fb318656433ddea1"
server_key_iv: "af427c3e012906823b62f8496d2e1a1a"
server_key_version: "encryptionkey@0"
I20260620 11:33:26.518512 5510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "2995a00c9f944adaa4c37d0cfa29d3aa"
format_stamp: "Formatted at 2026-06-20 11:33:26 on dist-test-slave-ctbw"
server_key: "4ceab65a6fa00c75fb318656433ddea1"
server_key_iv: "af427c3e012906823b62f8496d2e1a1a"
server_key_version: "encryptionkey@0"
I20260620 11:33:26.519834 5510 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:33:26.520507 5525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:26.520690 5510 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:33:26.520799 5510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "2995a00c9f944adaa4c37d0cfa29d3aa"
format_stamp: "Formatted at 2026-06-20 11:33:26 on dist-test-slave-ctbw"
server_key: "4ceab65a6fa00c75fb318656433ddea1"
server_key_iv: "af427c3e012906823b62f8496d2e1a1a"
server_key_version: "encryptionkey@0"
I20260620 11:33:26.520859 5510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:26.541164 5510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:26.543581 5510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:26.543699 5510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:26.543886 5510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:26.544250 5510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:26.544322 5510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:26.544358 5510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:26.544384 5510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:26.550984 5510 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:34035
I20260620 11:33:26.551064 5638 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:34035 every 8 connection(s)
I20260620 11:33:26.551369 5510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:33:26.553241 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 5510
I20260620 11:33:26.553339 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:33:26.553602 3022 external_mini_cluster.cc:1428] Setting key 66c09c70458a265fd11bac7c6917f48b
Jun 20 11:33:26 dist-test-slave-ctbw krb5kdc[5403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955206, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:26.562228 5641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:26.552069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43275 (local address 127.2.243.129:35351)
0620 11:33:26.552245 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:26.552280 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:26.552771 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:26.553013 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:26.553016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:26.553792 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:26.554044 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:26.554050 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:26.554931 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:26.554933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:26.556019 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:26.556022 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:26.556454 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:26.556455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:26.556572 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:26.557299 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:26.557309 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:26.559030 (+ 1721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:26.560892 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:26.560898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:26.560899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:26.561104 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:26.561342 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:26.561343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:26.561343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:26.561439 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:26.561761 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:26.561765 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:26.561870 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 11:33:26.561965 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 11:33:26.562076 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260620 11:33:26.562227 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:26.552075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:35351 (local address 127.2.243.190:43275)
0620 11:33:26.552124 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:26.552126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:26.552365 (+ 239us) server_negotiation.cc:407] Connection header received
0620 11:33:26.552864 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:26.552865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:26.552875 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:26.552931 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:26.554148 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:26.554780 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:26.556137 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:26.556360 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:26.556437 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:26.559129 (+ 2692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:26.559131 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:26.559131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:26.559140 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:26.560799 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:26.561182 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:26.561183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:26.561184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:26.561265 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:26.561510 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:26.561511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:26.561512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:26.561670 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:26.561772 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:26.562012 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 11:33:26.562103 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:33:26.562846 5639 heartbeater.cc:347] Connected to a master server at 127.2.243.190:43275
I20260620 11:33:26.562969 5639 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:26.563161 5639 heartbeater.cc:511] Master 127.2.243.190:43275 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.243.130@KRBTEST.COM; defaulting to no policy
I20260620 11:33:26.563709 5448 ts_manager.cc:194] Registered new tserver with Master: 2995a00c9f944adaa4c37d0cfa29d3aa (127.2.243.129:34035)
I20260620 11:33:26.564343 5448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:35351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
I20260620 11:33:26.581156 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:43275
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:42535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:26.662148 5644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:26.662391 5644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:26.662417 5644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:26.663877 5644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:26.663932 5644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:26.663961 5644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:33:26.665760 5644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:42535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:43275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:26.666086 5644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:26.666354 5644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:26.672878 5649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:26.672907 5652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:26.672894 5650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:26.673202 5644 server_base.cc:1034] running on GCE node
I20260620 11:33:26.673435 5644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:26.673678 5644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:26.674839 5644 hybrid_clock.cc:648] HybridClock initialized: now 1781955206674839 us; error 41 us; skew 500 ppm
Jun 20 11:33:26 dist-test-slave-ctbw krb5kdc[5403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955206, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:26.677371 5644 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:33:26.677847 5644 webserver.cc:460] Webserver started at http://127.2.243.130:33979/ using document root <none> and password file <none>
I20260620 11:33:26.678083 5644 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:26.678153 5644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:26.678287 5644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:26.679436 5644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "750344f81c5d460f9853fa0f4957f164"
format_stamp: "Formatted at 2026-06-20 11:33:26 on dist-test-slave-ctbw"
server_key: "50b970d24bb373e7a52af467719b1b83"
server_key_iv: "e2032d63670e325920e14a2d58f63f55"
server_key_version: "encryptionkey@0"
I20260620 11:33:26.679783 5644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "750344f81c5d460f9853fa0f4957f164"
format_stamp: "Formatted at 2026-06-20 11:33:26 on dist-test-slave-ctbw"
server_key: "50b970d24bb373e7a52af467719b1b83"
server_key_iv: "e2032d63670e325920e14a2d58f63f55"
server_key_version: "encryptionkey@0"
I20260620 11:33:26.681100 5644 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:33:26.681800 5659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:26.681967 5644 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:26.682062 5644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "750344f81c5d460f9853fa0f4957f164"
format_stamp: "Formatted at 2026-06-20 11:33:26 on dist-test-slave-ctbw"
server_key: "50b970d24bb373e7a52af467719b1b83"
server_key_iv: "e2032d63670e325920e14a2d58f63f55"
server_key_version: "encryptionkey@0"
I20260620 11:33:26.682137 5644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:26.705000 5644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:26.707585 5644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:26.707744 5644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:26.707978 5644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:26.708315 5644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:26.708367 5644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:26.708412 5644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:26.708438 5644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:26.714854 5644 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:41221
I20260620 11:33:26.714972 5772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:41221 every 8 connection(s)
I20260620 11:33:26.715363 5644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:33:26.715879 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 5644
I20260620 11:33:26.715981 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:33:26.716216 3022 external_mini_cluster.cc:1428] Setting key 7a935af8619959cd8f00de4d5bb131a9
Jun 20 11:33:26 dist-test-slave-ctbw krb5kdc[5403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955206, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.243.131@KRBTEST.COM; defaulting to no policy
I20260620 11:33:26.726353 5487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:26.716338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:49599 (local address 127.2.243.190:43275)
0620 11:33:26.716413 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:26.716415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:26.716703 (+ 288us) server_negotiation.cc:407] Connection header received
0620 11:33:26.717495 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:26.717497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:26.717508 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:26.717571 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:26.718610 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:26.719194 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:26.720765 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:26.720977 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:26.721033 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:26.723470 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:26.723471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:26.723472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:26.723479 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:26.725011 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:26.725403 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:26.725404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:26.725404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:26.725461 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:26.725703 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:26.725704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:26.725705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:26.725816 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:26.725912 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:26.726167 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 11:33:26.726245 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 11:33:26.726554 5775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:26.716359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43275 (local address 127.2.243.130:49599)
0620 11:33:26.716550 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:26.716598 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:26.717371 (+ 773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:26.717642 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:26.717646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:26.718302 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:26.718507 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:26.718511 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:26.719282 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:26.719283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:26.720666 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:26.720669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:26.721057 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:26.721058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:26.721209 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:26.721735 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:26.721746 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:26.723373 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:26.725098 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:26.725105 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:26.725107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:26.725328 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:26.725524 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:26.725525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:26.725525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:26.725638 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:26.725886 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:26.725891 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:26.726041 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 11:33:26.726148 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 11:33:26.726310 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":94,"threads_started":1}
I20260620 11:33:26.727053 5773 heartbeater.cc:347] Connected to a master server at 127.2.243.190:43275
I20260620 11:33:26.727162 5773 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:26.727339 5773 heartbeater.cc:511] Master 127.2.243.190:43275 requested a full tablet report, sending...
I20260620 11:33:26.727732 5448 ts_manager.cc:194] Registered new tserver with Master: 750344f81c5d460f9853fa0f4957f164 (127.2.243.130:41221)
I20260620 11:33:26.728116 5448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:49599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
W20260620 11:33:27.499327 3022 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.2.243.131: real 0.782s user 0.000s sys 0.003s
I20260620 11:33:27.499750 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:43275
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:42535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:33:27.565228 5639 heartbeater.cc:503] Master 127.2.243.190:43275 was elected leader, sending a full tablet report...
W20260620 11:33:27.582664 5778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:27.582903 5778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:27.582928 5778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:27.584123 5778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:27.584175 5778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:27.584203 5778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:33:27.585750 5778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:42535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:43275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:27.586017 5778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:27.586261 5778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:27.591316 5784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:27.591355 5783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:27.591292 5786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:27.591634 5778 server_base.cc:1034] running on GCE node
I20260620 11:33:27.591856 5778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:27.592096 5778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:27.593257 5778 hybrid_clock.cc:648] HybridClock initialized: now 1781955207593239 us; error 34 us; skew 500 ppm
Jun 20 11:33:27 dist-test-slave-ctbw krb5kdc[5403](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955207, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:27.595825 5778 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:33:27.596305 5778 webserver.cc:460] Webserver started at http://127.2.243.131:40183/ using document root <none> and password file <none>
I20260620 11:33:27.596535 5778 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:27.596597 5778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:27.596730 5778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:27.597784 5778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "8d5f695e8ebd46c684b197001af4b839"
format_stamp: "Formatted at 2026-06-20 11:33:27 on dist-test-slave-ctbw"
server_key: "03bc33b3217eef5de77277413af599c5"
server_key_iv: "8f47892b39e53246678e9ea97bb2c6ed"
server_key_version: "encryptionkey@0"
I20260620 11:33:27.598142 5778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "8d5f695e8ebd46c684b197001af4b839"
format_stamp: "Formatted at 2026-06-20 11:33:27 on dist-test-slave-ctbw"
server_key: "03bc33b3217eef5de77277413af599c5"
server_key_iv: "8f47892b39e53246678e9ea97bb2c6ed"
server_key_version: "encryptionkey@0"
I20260620 11:33:27.599685 5778 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:33:27.600569 5793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:27.600760 5778 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:27.600849 5778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "8d5f695e8ebd46c684b197001af4b839"
format_stamp: "Formatted at 2026-06-20 11:33:27 on dist-test-slave-ctbw"
server_key: "03bc33b3217eef5de77277413af599c5"
server_key_iv: "8f47892b39e53246678e9ea97bb2c6ed"
server_key_version: "encryptionkey@0"
I20260620 11:33:27.600922 5778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:27.627564 5778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:27.629667 5778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:27.629801 5778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:27.630008 5778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:27.630268 5778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:27.630301 5778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:27.630326 5778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:27.630342 5778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:27.636915 5778 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:46741
I20260620 11:33:27.636988 5906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:46741 every 8 connection(s)
I20260620 11:33:27.637344 5778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
Jun 20 11:33:27 dist-test-slave-ctbw krb5kdc[5403](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955207, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:27.645459 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 5778
I20260620 11:33:27.645601 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:33:27.645848 3022 external_mini_cluster.cc:1428] Setting key 299619990b54c577cd585d6b10dfb3ef
I20260620 11:33:27.647007 5910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:27.638172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:56755 (local address 127.2.243.190:43275)
0620 11:33:27.638298 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:27.638300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:27.638469 (+ 169us) server_negotiation.cc:407] Connection header received
0620 11:33:27.638979 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:27.638980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:27.638995 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:27.639063 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:27.640022 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:27.640543 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:27.641530 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:27.641722 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:27.641780 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:27.644069 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:27.644070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:27.644070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:27.644077 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:27.645444 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:27.645939 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:27.645940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:27.645940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:27.646004 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:27.646289 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:27.646290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:27.646291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:27.646416 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:27.646478 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:27.646771 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 11:33:27.646911 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":76,"threads_started":1}
I20260620 11:33:27.647172 5909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:27.638133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43275 (local address 127.2.243.131:56755)
0620 11:33:27.638297 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:27.638338 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:27.638891 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:27.639127 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:27.639130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:27.639722 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:27.639922 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:27.639927 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:27.640629 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:27.640629 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:27.641415 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:27.641416 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:27.641814 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:27.641815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:27.641928 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:27.642487 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:27.642495 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:27.643979 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:27.645527 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:27.645534 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:27.645536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:27.645842 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:27.646069 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:27.646070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:27.646071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:27.646221 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:27.646472 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:27.646478 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:27.646633 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 11:33:27.646745 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 11:33:27.646971 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260620 11:33:27.647624 5907 heartbeater.cc:347] Connected to a master server at 127.2.243.190:43275
I20260620 11:33:27.647722 5907 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:27.647891 5907 heartbeater.cc:511] Master 127.2.243.190:43275 requested a full tablet report, sending...
I20260620 11:33:27.648272 5448 ts_manager.cc:194] Registered new tserver with Master: 8d5f695e8ebd46c684b197001af4b839 (127.2.243.131:46741)
I20260620 11:33:27.648699 5448 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:56755
I20260620 11:33:27.657624 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 11:33:27.668494 5910 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:27.667722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.2.243.190:43275)
0620 11:33:27.667788 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:27.667790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:27.667797 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:33:27.667818 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:27.667818 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:27.667826 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:27.667887 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:27.668412 (+ 525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52454: BlockingRecv error: recv got EOF from 127.0.0.1:52454 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 11:33:27.669317 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 5510
I20260620 11:33:27.673719 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 5644
I20260620 11:33:27.678056 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 5778
I20260620 11:33:27.682247 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 5416
2026-06-20T11:33:27Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2652 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:33:30 dist-test-slave-ctbw krb5kdc[5920](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:33:30 dist-test-slave-ctbw krb5kdc[5920](info): set up 2 sockets
Jun 20 11:33:30 dist-test-slave-ctbw krb5kdc[5920](info): commencing operation
krb5kdc: starting...
W20260620 11:33:31.010282 3022 mini_kdc.cc:121] Time spent starting KDC: real 3.307s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:33:31 dist-test-slave-ctbw krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:33:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:33:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:33:31.165081 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:46473
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:35457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:46473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:31.247145 5933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:31.247365 5933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:31.247390 5933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:31.248540 5933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:33:31.248586 5933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:31.248603 5933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:31.248618 5933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:33:31.248673 5933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:33:31.250501 5933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:35457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:46473
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:46473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:31.250756 5933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:31.251019 5933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:31.255569 5938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:31.255568 5941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:31.255522 5939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:31.255784 5933 server_base.cc:1034] running on GCE node
I20260620 11:33:31.256006 5933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:31.256256 5933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:31.257387 5933 hybrid_clock.cc:648] HybridClock initialized: now 1781955211257375 us; error 27 us; skew 500 ppm
Jun 20 11:33:31 dist-test-slave-ctbw krb5kdc[5920](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955211, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:31.262794 5933 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:33:31.263271 5933 webserver.cc:460] Webserver started at http://127.2.243.190:33641/ using document root <none> and password file <none>
I20260620 11:33:31.263492 5933 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:31.263543 5933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:31.263695 5933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:31.264726 5933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "57c98510e19d47288bb9685f1bd2d3d8"
format_stamp: "Formatted at 2026-06-20 11:33:31 on dist-test-slave-ctbw"
server_key: "cd906578d12aaf595151a9eabd574171"
server_key_iv: "12e64f6717b890900c616398644def09"
server_key_version: "encryptionkey@0"
I20260620 11:33:31.265075 5933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "57c98510e19d47288bb9685f1bd2d3d8"
format_stamp: "Formatted at 2026-06-20 11:33:31 on dist-test-slave-ctbw"
server_key: "cd906578d12aaf595151a9eabd574171"
server_key_iv: "12e64f6717b890900c616398644def09"
server_key_version: "encryptionkey@0"
I20260620 11:33:31.266383 5933 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:33:31.267246 5948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:31.267417 5933 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 11:33:31.267508 5933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "57c98510e19d47288bb9685f1bd2d3d8"
format_stamp: "Formatted at 2026-06-20 11:33:31 on dist-test-slave-ctbw"
server_key: "cd906578d12aaf595151a9eabd574171"
server_key_iv: "12e64f6717b890900c616398644def09"
server_key_version: "encryptionkey@0"
I20260620 11:33:31.267589 5933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:31.273900 5933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:31.275785 5933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:31.275925 5933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:31.279856 5933 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:46473
I20260620 11:33:31.279958 6000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:46473 every 8 connection(s)
I20260620 11:33:31.280225 5933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:33:31.281051 6001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:31.283398 6001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8: Bootstrap starting.
I20260620 11:33:31.283977 6001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:31.284269 6001 log.cc:824] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:31.284984 6001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8: No bootstrap required, opened a new log
I20260620 11:33:31.286609 6001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c98510e19d47288bb9685f1bd2d3d8" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46473 } }
I20260620 11:33:31.286748 6001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:31.286805 6001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57c98510e19d47288bb9685f1bd2d3d8, State: Initialized, Role: FOLLOWER
I20260620 11:33:31.286983 6001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c98510e19d47288bb9685f1bd2d3d8" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46473 } }
I20260620 11:33:31.287051 6001 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:33:31.287110 6001 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:33:31.287163 6001 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:31.288014 6001 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c98510e19d47288bb9685f1bd2d3d8" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46473 } }
I20260620 11:33:31.288130 6001 leader_election.cc:302] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57c98510e19d47288bb9685f1bd2d3d8; no voters:
I20260620 11:33:31.288304 6001 leader_election.cc:288] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:33:31.288344 6004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:31.288503 6004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8 [term 1 LEADER]: Becoming Leader. State: Replica: 57c98510e19d47288bb9685f1bd2d3d8, State: Running, Role: LEADER
I20260620 11:33:31.288606 6001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:31.288643 6004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c98510e19d47288bb9685f1bd2d3d8" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46473 } }
I20260620 11:33:31.289091 6005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57c98510e19d47288bb9685f1bd2d3d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c98510e19d47288bb9685f1bd2d3d8" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46473 } } }
I20260620 11:33:31.289099 6006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57c98510e19d47288bb9685f1bd2d3d8. Latest consensus state: current_term: 1 leader_uuid: "57c98510e19d47288bb9685f1bd2d3d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57c98510e19d47288bb9685f1bd2d3d8" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46473 } } }
I20260620 11:33:31.289289 6006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8 [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:31.289531 6005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8 [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:31.289587 6013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:33:31.289933 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 5933
I20260620 11:33:31.290026 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:33:31.290135 6013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:33:31.290225 3022 external_mini_cluster.cc:1428] Setting key e7ba4f52fb0085737b7b83c0977d6b5b
I20260620 11:33:31.291829 6013 catalog_manager.cc:1347] Generated new cluster ID: 5a380ca085914f15b5d102d2061ee48d
I20260620 11:33:31.291883 6013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:33:31 dist-test-slave-ctbw krb5kdc[5920](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955211, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:31.302361 6023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:31.290828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59370 (local address 127.2.243.190:46473)
0620 11:33:31.291010 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:31.291019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:31.291225 (+ 206us) server_negotiation.cc:407] Connection header received
0620 11:33:31.291508 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:31.291516 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:31.291576 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:31.292478 (+ 902us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:31.292568 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:31.293929 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:31.295020 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:31.295684 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:31.295809 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:31.298484 (+ 2675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:31.298488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:31.298489 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:31.298502 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:31.300546 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:31.301005 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:31.301008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:31.301008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:31.301119 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:31.301338 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:31.301338 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:31.301339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:31.301634 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:31.301816 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:31.301884 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 11:33:31.302014 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":106,"threads_started":1}
I20260620 11:33:31.303565 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:33:31.319288 6013 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:33:31.320122 6013 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:33:31.328613 6023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:31.324290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59376 (local address 127.2.243.190:46473)
0620 11:33:31.324345 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:31.324347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:31.324514 (+ 167us) server_negotiation.cc:407] Connection header received
0620 11:33:31.324594 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:31.324595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:31.324608 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:31.324658 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:31.325125 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:31.325865 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:31.326952 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:31.327248 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:31.327310 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:31.328075 (+ 765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:31.328076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:31.328077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:33:31.328322 (+ 245us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:31.328343 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:33:31.328349 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:31.328407 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:31.328497 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 11:33:31.328524 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 11:33:31.329411 6013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57c98510e19d47288bb9685f1bd2d3d8: Generated new TSK 0
I20260620 11:33:31.329592 6013 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:33:31.334793 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 5933
2026-06-20T11:33:31Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (3651 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:33:31 dist-test-slave-ctbw krb5kdc[6034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:33:31 dist-test-slave-ctbw krb5kdc[6034](info): set up 2 sockets
Jun 20 11:33:31 dist-test-slave-ctbw krb5kdc[6034](info): commencing operation
krb5kdc: starting...
W20260620 11:33:32.371106 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:33:32 dist-test-slave-ctbw krb5kdc[6034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:33:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:33:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:33:32.491882 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:33233
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:46103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:33233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:32.573235 6047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:32.573503 6047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:32.573551 6047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:32.574734 6047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:33:32.574803 6047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:32.574827 6047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:32.574848 6047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:33:32.574895 6047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:33:32.576462 6047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:46103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:33233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:33233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:32.576740 6047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:32.576965 6047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:32.581916 6052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:32.581938 6053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:32.581959 6055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:32.582285 6047 server_base.cc:1034] running on GCE node
I20260620 11:33:32.582521 6047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:32.582821 6047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:32.583998 6047 hybrid_clock.cc:648] HybridClock initialized: now 1781955212583976 us; error 41 us; skew 500 ppm
Jun 20 11:33:32 dist-test-slave-ctbw krb5kdc[6034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955212, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:32.586385 6047 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:33:32.586844 6047 webserver.cc:460] Webserver started at http://127.2.243.190:45157/ using document root <none> and password file <none>
I20260620 11:33:32.587117 6047 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:32.587164 6047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:32.587296 6047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:32.588356 6047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "8bad8a23ee4640d18bd0ede16833b9eb"
format_stamp: "Formatted at 2026-06-20 11:33:32 on dist-test-slave-ctbw"
server_key: "88de2bc597dea7083b02a6a90881ff03"
server_key_iv: "eeb55e42842c70c8132974fbc2e72be1"
server_key_version: "encryptionkey@0"
I20260620 11:33:32.588722 6047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "8bad8a23ee4640d18bd0ede16833b9eb"
format_stamp: "Formatted at 2026-06-20 11:33:32 on dist-test-slave-ctbw"
server_key: "88de2bc597dea7083b02a6a90881ff03"
server_key_iv: "eeb55e42842c70c8132974fbc2e72be1"
server_key_version: "encryptionkey@0"
I20260620 11:33:32.590165 6047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:33:32.591003 6062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:32.591176 6047 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:32.591272 6047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "8bad8a23ee4640d18bd0ede16833b9eb"
format_stamp: "Formatted at 2026-06-20 11:33:32 on dist-test-slave-ctbw"
server_key: "88de2bc597dea7083b02a6a90881ff03"
server_key_iv: "eeb55e42842c70c8132974fbc2e72be1"
server_key_version: "encryptionkey@0"
I20260620 11:33:32.591351 6047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:32.609459 6047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:32.611622 6047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:32.611788 6047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:32.615717 6047 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:33233
I20260620 11:33:32.615764 6114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:33233 every 8 connection(s)
I20260620 11:33:32.616119 6047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:33:32.616771 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 6047
I20260620 11:33:32.616859 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:33:32.617044 3022 external_mini_cluster.cc:1428] Setting key a2f401efbdf48d2211288c8322abd529
I20260620 11:33:32.617197 6115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:32.620654 6115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb: Bootstrap starting.
I20260620 11:33:32.621343 6115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:32.621649 6115 log.cc:824] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:32.622375 6115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb: No bootstrap required, opened a new log
Jun 20 11:33:32 dist-test-slave-ctbw krb5kdc[6034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:32.624490 6115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bad8a23ee4640d18bd0ede16833b9eb" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33233 } }
I20260620 11:33:32.624648 6115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:32.624697 6115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bad8a23ee4640d18bd0ede16833b9eb, State: Initialized, Role: FOLLOWER
I20260620 11:33:32.624833 6115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bad8a23ee4640d18bd0ede16833b9eb" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33233 } }
I20260620 11:33:32.624907 6115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:33:32.624928 6115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:33:32.624989 6115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:32.625648 6115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bad8a23ee4640d18bd0ede16833b9eb" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33233 } }
I20260620 11:33:32.625759 6115 leader_election.cc:302] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bad8a23ee4640d18bd0ede16833b9eb; no voters:
I20260620 11:33:32.625950 6115 leader_election.cc:288] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:33:32.626020 6120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:32.626204 6120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb [term 1 LEADER]: Becoming Leader. State: Replica: 8bad8a23ee4640d18bd0ede16833b9eb, State: Running, Role: LEADER
I20260620 11:33:32.626207 6115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:32.626310 6120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bad8a23ee4640d18bd0ede16833b9eb" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33233 } }
I20260620 11:33:32.627640 6122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bad8a23ee4640d18bd0ede16833b9eb. Latest consensus state: current_term: 1 leader_uuid: "8bad8a23ee4640d18bd0ede16833b9eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bad8a23ee4640d18bd0ede16833b9eb" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33233 } } }
I20260620 11:33:32.627816 6121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bad8a23ee4640d18bd0ede16833b9eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bad8a23ee4640d18bd0ede16833b9eb" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33233 } } }
I20260620 11:33:32.627915 6122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:32.628041 6121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb [sys.catalog]: This master's current role is: LEADER
W20260620 11:33:32.628767 6134 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:33:32.628820 6134 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:33:32.628883 6135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:33:32.629040 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:32.617539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46958 (local address 127.2.243.190:33233)
0620 11:33:32.617697 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:32.617707 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:32.617739 (+ 32us) server_negotiation.cc:407] Connection header received
0620 11:33:32.618306 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:32.618311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:32.618355 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:32.618465 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:32.619323 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:32.620249 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:32.621315 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:32.621803 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:32.621925 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:32.624722 (+ 2797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:32.624726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:32.624726 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:32.624739 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:32.626955 (+ 2216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:32.627339 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:32.627342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:32.627342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:32.627470 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:32.627709 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:32.627710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:32.627710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:32.628030 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:32.628194 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:32.628665 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 11:33:32.628782 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260620 11:33:32.629074 6135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:33:32.630450 6135 catalog_manager.cc:1347] Generated new cluster ID: ed38977a7fe34229b6f7bda21c0ab6a2
I20260620 11:33:32.630501 6135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:33:32.635025 6135 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:33:32.635499 6135 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:33:32.646688 6135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bad8a23ee4640d18bd0ede16833b9eb: Generated new TSK 0
I20260620 11:33:32.646910 6135 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
I20260620 11:33:32.679507 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33233
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:46103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:32.761793 6141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:32.762003 6141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:32.762034 6141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:32.763123 6141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:32.763159 6141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:32.763185 6141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:33:32.764600 6141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:46103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:32.764779 6141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:32.764948 6141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:32.769840 6146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:32.769886 6149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:32.769905 6147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:32.769948 6141 server_base.cc:1034] running on GCE node
I20260620 11:33:32.770279 6141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:32.770514 6141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:32.771670 6141 hybrid_clock.cc:648] HybridClock initialized: now 1781955212771660 us; error 44 us; skew 500 ppm
Jun 20 11:33:32 dist-test-slave-ctbw krb5kdc[6034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955212, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:32.774205 6141 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:33:32.774696 6141 webserver.cc:460] Webserver started at http://127.2.243.129:37539/ using document root <none> and password file <none>
I20260620 11:33:32.774967 6141 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:32.775032 6141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:32.775230 6141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:32.776265 6141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "40e7bb889e3341989c7d44309abd2587"
format_stamp: "Formatted at 2026-06-20 11:33:32 on dist-test-slave-ctbw"
server_key: "efcdfb09916cb6b496623d919b670eff"
server_key_iv: "347bc05844fdeb079de7cdaeff52be30"
server_key_version: "encryptionkey@0"
I20260620 11:33:32.776635 6141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "40e7bb889e3341989c7d44309abd2587"
format_stamp: "Formatted at 2026-06-20 11:33:32 on dist-test-slave-ctbw"
server_key: "efcdfb09916cb6b496623d919b670eff"
server_key_iv: "347bc05844fdeb079de7cdaeff52be30"
server_key_version: "encryptionkey@0"
I20260620 11:33:32.777963 6141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:33:32.778739 6156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:32.778928 6141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:33:32.779032 6141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "40e7bb889e3341989c7d44309abd2587"
format_stamp: "Formatted at 2026-06-20 11:33:32 on dist-test-slave-ctbw"
server_key: "efcdfb09916cb6b496623d919b670eff"
server_key_iv: "347bc05844fdeb079de7cdaeff52be30"
server_key_version: "encryptionkey@0"
I20260620 11:33:32.779150 6141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:32.796658 6141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:32.798734 6141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:32.798939 6141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:32.799214 6141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:32.799520 6141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:32.799575 6141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:32.799616 6141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:32.799643 6141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:32.805536 6141 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:45739
I20260620 11:33:32.805590 6269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:45739 every 8 connection(s)
I20260620 11:33:32.805883 6141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
Jun 20 11:33:32 dist-test-slave-ctbw krb5kdc[6034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955212, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:32.814298 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 6141
I20260620 11:33:32.814389 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:33:32.814690 3022 external_mini_cluster.cc:1428] Setting key c5e7d123bb469c9ebc4817bbb14d24d5
I20260620 11:33:32.815986 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:32.806566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:57825 (local address 127.2.243.190:33233)
0620 11:33:32.806612 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:32.806614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:32.806837 (+ 223us) server_negotiation.cc:407] Connection header received
0620 11:33:32.807380 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:32.807381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:32.807390 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:32.807450 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:32.808498 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:32.809012 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:32.810147 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:32.810349 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:32.810412 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:32.812831 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:32.812833 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:32.812833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:32.812839 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:32.814402 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:32.814960 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:32.814961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:32.814962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:32.815034 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:32.815312 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:32.815313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:32.815313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:32.815432 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:32.815520 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:32.815795 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 11:33:32.815891 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:33:32.816215 6272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:32.806538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33233 (local address 127.2.243.129:57825)
0620 11:33:32.806715 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:32.806761 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:32.807294 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:32.807529 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:32.807533 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:32.808192 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:32.808399 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:32.808405 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:32.809123 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:32.809124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:32.810002 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:32.810004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:32.810462 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:32.810463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:32.810604 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:32.811252 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:32.811261 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:32.812705 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:32.814517 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:32.814525 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:32.814527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:32.814846 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:32.815121 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:32.815122 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:32.815122 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:32.815234 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:32.815500 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:32.815507 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:32.815646 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 11:33:32.815754 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 11:33:32.815914 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260620 11:33:32.816740 6270 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33233
I20260620 11:33:32.816836 6270 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:32.817032 6270 heartbeater.cc:511] Master 127.2.243.190:33233 requested a full tablet report, sending...
I20260620 11:33:32.817610 6079 ts_manager.cc:194] Registered new tserver with Master: 40e7bb889e3341989c7d44309abd2587 (127.2.243.129:45739)
I20260620 11:33:32.818454 6079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:57825
WARNING: no policy specified for kudu/127.2.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
I20260620 11:33:32.840071 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33233
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:46103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:32.921499 6275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:32.921706 6275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:32.921729 6275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:32.922804 6275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:32.922840 6275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:32.922909 6275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:33:32.924336 6275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:46103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:32.924530 6275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:32.924717 6275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:32.929672 6283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:32.929713 6281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:32.929677 6280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:32.929874 6275 server_base.cc:1034] running on GCE node
I20260620 11:33:32.930163 6275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:32.930425 6275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:32.931605 6275 hybrid_clock.cc:648] HybridClock initialized: now 1781955212931595 us; error 37 us; skew 500 ppm
Jun 20 11:33:32 dist-test-slave-ctbw krb5kdc[6034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955212, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:32.934199 6275 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:33:32.934723 6275 webserver.cc:460] Webserver started at http://127.2.243.130:39225/ using document root <none> and password file <none>
I20260620 11:33:32.935010 6275 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:32.935091 6275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:32.935238 6275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:32.936333 6275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "1410a5bc99bf45739ccc6302086f4dba"
format_stamp: "Formatted at 2026-06-20 11:33:32 on dist-test-slave-ctbw"
server_key: "9f3bf80627fabc3f9861412f3f3c3685"
server_key_iv: "f1518f4cc90dbceb3caa0b0e224e01d6"
server_key_version: "encryptionkey@0"
I20260620 11:33:32.936726 6275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "1410a5bc99bf45739ccc6302086f4dba"
format_stamp: "Formatted at 2026-06-20 11:33:32 on dist-test-slave-ctbw"
server_key: "9f3bf80627fabc3f9861412f3f3c3685"
server_key_iv: "f1518f4cc90dbceb3caa0b0e224e01d6"
server_key_version: "encryptionkey@0"
I20260620 11:33:32.938189 6275 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:33:32.939044 6290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:32.939239 6275 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:32.939347 6275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "1410a5bc99bf45739ccc6302086f4dba"
format_stamp: "Formatted at 2026-06-20 11:33:32 on dist-test-slave-ctbw"
server_key: "9f3bf80627fabc3f9861412f3f3c3685"
server_key_iv: "f1518f4cc90dbceb3caa0b0e224e01d6"
server_key_version: "encryptionkey@0"
I20260620 11:33:32.939427 6275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:32.964859 6275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:32.966886 6275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:32.966995 6275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:32.967166 6275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:32.967478 6275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:32.967515 6275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:32.967567 6275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:32.967603 6275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:32.973392 6275 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:43033
I20260620 11:33:32.973479 6403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:43033 every 8 connection(s)
I20260620 11:33:32.973809 6275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:33:32.975135 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 6275
I20260620 11:33:32.975246 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:33:32.975483 3022 external_mini_cluster.cc:1428] Setting key b511d22c0dd09615b24b6b0515161caf
Jun 20 11:33:32 dist-test-slave-ctbw krb5kdc[6034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955212, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.243.131@KRBTEST.COM; defaulting to no policy
I20260620 11:33:32.984218 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:32.974559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:41885 (local address 127.2.243.190:33233)
0620 11:33:32.974644 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:32.974646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:32.974840 (+ 194us) server_negotiation.cc:407] Connection header received
0620 11:33:32.975477 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:32.975479 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:32.975492 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:32.975568 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:32.976535 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:32.977332 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:32.978421 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:32.978634 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:32.978719 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:32.981120 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:32.981122 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:32.981122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:32.981130 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:32.982764 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:32.983233 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:32.983235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:32.983235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:32.983300 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:32.983560 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:32.983561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:32.983561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:32.983685 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:32.983758 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:32.984037 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 11:33:32.984117 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 11:33:32.984439 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:32.974531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33233 (local address 127.2.243.130:41885)
0620 11:33:32.974716 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:32.974754 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:32.975377 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:32.975654 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:32.975658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:32.976235 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:32.976432 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:32.976437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:32.977459 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:32.977460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:32.978330 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:32.978332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:32.978731 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:32.978732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:32.978919 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:32.979626 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:32.979634 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:32.980995 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:32.982874 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:32.982881 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:32.982884 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:32.983151 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:32.983362 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:32.983363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:32.983363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:32.983500 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:32.983760 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:32.983766 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:32.983907 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 11:33:32.984006 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 11:33:32.984198 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":112,"threads_started":1}
I20260620 11:33:32.984990 6404 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33233
I20260620 11:33:32.985085 6404 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:32.985344 6404 heartbeater.cc:511] Master 127.2.243.190:33233 requested a full tablet report, sending...
I20260620 11:33:32.985725 6079 ts_manager.cc:194] Registered new tserver with Master: 1410a5bc99bf45739ccc6302086f4dba (127.2.243.130:43033)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.131@KRBTEST.COM" created.
I20260620 11:33:32.986253 6079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:41885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
I20260620 11:33:33.001672 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33233
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:46103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:33.082957 6409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:33.083155 6409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:33.083178 6409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:33.084285 6409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:33.084321 6409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:33.084347 6409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:33:33.085685 6409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:46103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:33.085857 6409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:33.086081 6409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:33.090894 6415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:33.091032 6417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:33.090957 6414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:33.091295 6409 server_base.cc:1034] running on GCE node
I20260620 11:33:33.091470 6409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:33.091677 6409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:33.092813 6409 hybrid_clock.cc:648] HybridClock initialized: now 1781955213092794 us; error 35 us; skew 500 ppm
Jun 20 11:33:33 dist-test-slave-ctbw krb5kdc[6034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955213, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:33.095343 6409 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:33:33.095820 6409 webserver.cc:460] Webserver started at http://127.2.243.131:44537/ using document root <none> and password file <none>
I20260620 11:33:33.096073 6409 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:33.096139 6409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:33.096288 6409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:33.097409 6409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "0896cdbe62a94435ba4ef29bb5ea14ae"
format_stamp: "Formatted at 2026-06-20 11:33:33 on dist-test-slave-ctbw"
server_key: "659b385428fba6f4902f8cc8179cec75"
server_key_iv: "cd4cc89bee69ecc00bf113847a356925"
server_key_version: "encryptionkey@0"
I20260620 11:33:33.097774 6409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "0896cdbe62a94435ba4ef29bb5ea14ae"
format_stamp: "Formatted at 2026-06-20 11:33:33 on dist-test-slave-ctbw"
server_key: "659b385428fba6f4902f8cc8179cec75"
server_key_iv: "cd4cc89bee69ecc00bf113847a356925"
server_key_version: "encryptionkey@0"
I20260620 11:33:33.099107 6409 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 11:33:33.099862 6424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:33.100028 6409 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:33.100123 6409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "0896cdbe62a94435ba4ef29bb5ea14ae"
format_stamp: "Formatted at 2026-06-20 11:33:33 on dist-test-slave-ctbw"
server_key: "659b385428fba6f4902f8cc8179cec75"
server_key_iv: "cd4cc89bee69ecc00bf113847a356925"
server_key_version: "encryptionkey@0"
I20260620 11:33:33.100198 6409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:33.137250 6409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:33.139683 6409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:33.139853 6409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:33.140105 6409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:33.140405 6409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:33.140455 6409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:33.140496 6409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:33.140527 6409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:33.146339 6409 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:39613
I20260620 11:33:33.146484 6537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:39613 every 8 connection(s)
I20260620 11:33:33.146823 6409 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:33:33.146965 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 6409
I20260620 11:33:33.147073 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:33:33.147277 3022 external_mini_cluster.cc:1428] Setting key 4fb1127e02d18cdeba05a6e23db6c65f
Jun 20 11:33:33 dist-test-slave-ctbw krb5kdc[6034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955213, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:33.156571 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:33.147633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:45513 (local address 127.2.243.190:33233)
0620 11:33:33.147677 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:33.147680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:33.147995 (+ 315us) server_negotiation.cc:407] Connection header received
0620 11:33:33.148525 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:33.148526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:33.148536 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:33.148584 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:33.149560 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:33.150115 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:33.151173 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:33.151363 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:33.151433 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:33.153580 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:33.153581 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:33.153582 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:33.153588 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:33.155194 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:33.155569 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:33.155571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:33.155571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:33.155634 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:33.155894 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:33.155895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:33.155896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:33.156018 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:33.156083 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:33.156430 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 11:33:33.156502 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:33:33.156836 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:33.147650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33233 (local address 127.2.243.131:45513)
0620 11:33:33.147889 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:33.147934 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:33.148441 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:33.148638 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:33.148641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:33.149197 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:33.149396 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:33.149400 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:33.150201 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:33.150202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:33.151096 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:33.151099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:33.151442 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:33.151443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:33.151540 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:33.152074 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:33.152082 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:33.153458 (+ 1376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:33.155266 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:33.155272 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:33.155274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:33.155481 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:33.155700 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:33.155700 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:33.155701 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:33.155796 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:33.156174 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:33.156180 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:33.156304 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 11:33:33.156404 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 11:33:33.156543 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":89,"threads_started":1}
I20260620 11:33:33.157552 6538 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33233
I20260620 11:33:33.157677 6538 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:33.157860 6538 heartbeater.cc:511] Master 127.2.243.190:33233 requested a full tablet report, sending...
I20260620 11:33:33.158248 6079 ts_manager.cc:194] Registered new tserver with Master: 0896cdbe62a94435ba4ef29bb5ea14ae (127.2.243.131:39613)
I20260620 11:33:33.158747 6079 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:45513
I20260620 11:33:33.159034 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 11:33:33 dist-test-slave-ctbw krb5kdc[6034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955213, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 11:33:33 dist-test-slave-ctbw krb5kdc[6034](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955213, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:33.182271 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:33.174902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46970 (local address 127.2.243.190:33233)
0620 11:33:33.174972 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:33.174975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:33.175072 (+ 97us) server_negotiation.cc:407] Connection header received
0620 11:33:33.175215 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:33.175216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:33.175226 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:33.175296 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:33.175864 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:33.176358 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:33.177082 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:33.177274 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:33.177352 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:33.179710 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:33.179711 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:33.179712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:33.179720 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:33.181123 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:33.181468 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:33.181469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:33.181470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:33.181528 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:33.181729 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:33.181730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:33.181730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:33.181883 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:33.181983 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:33.182083 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 11:33:33.182165 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 11:33:33.182503 6078 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46970
I20260620 11:33:33.182960 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 6141
I20260620 11:33:33.187119 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 6275
I20260620 11:33:33.191282 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 6409
I20260620 11:33:33.195708 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 6047
2026-06-20T11:33:33Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1862 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:33:33 dist-test-slave-ctbw krb5kdc[6550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:33:33 dist-test-slave-ctbw krb5kdc[6550](info): set up 2 sockets
Jun 20 11:33:33 dist-test-slave-ctbw krb5kdc[6550](info): commencing operation
krb5kdc: starting...
W20260620 11:33:34.243126 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:33:34 dist-test-slave-ctbw krb5kdc[6550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:33:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:33:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:33:34.367862 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:45935
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:45935
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:34.450856 6563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:34.451119 6563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:34.451159 6563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:34.452311 6563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:33:34.452380 6563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:34.452409 6563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:34.452440 6563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:33:34.452471 6563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:33:34.453986 6563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:45935
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:45935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:34.454272 6563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:34.454493 6563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:34.459558 6571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:34.459551 6569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:34.459546 6568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:34.459617 6563 server_base.cc:1034] running on GCE node
I20260620 11:33:34.460047 6563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:34.460346 6563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:34.461495 6563 hybrid_clock.cc:648] HybridClock initialized: now 1781955214461482 us; error 30 us; skew 500 ppm
Jun 20 11:33:34 dist-test-slave-ctbw krb5kdc[6550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955214, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:34.464421 6563 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:33:34.464991 6563 webserver.cc:460] Webserver started at http://127.2.243.190:43277/ using document root <none> and password file <none>
I20260620 11:33:34.465217 6563 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:34.465263 6563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:34.465422 6563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:34.466470 6563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "5de1259f8067482cbcfaa5dd9f2c0000"
format_stamp: "Formatted at 2026-06-20 11:33:34 on dist-test-slave-ctbw"
server_key: "6e21919af7d093c6f6affd6bd07ea78f"
server_key_iv: "7cdff2767f60f7947662a9f0baef5e30"
server_key_version: "encryptionkey@0"
I20260620 11:33:34.466823 6563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "5de1259f8067482cbcfaa5dd9f2c0000"
format_stamp: "Formatted at 2026-06-20 11:33:34 on dist-test-slave-ctbw"
server_key: "6e21919af7d093c6f6affd6bd07ea78f"
server_key_iv: "7cdff2767f60f7947662a9f0baef5e30"
server_key_version: "encryptionkey@0"
I20260620 11:33:34.468155 6563 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:34.468951 6578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:34.469120 6563 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:34.469208 6563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "5de1259f8067482cbcfaa5dd9f2c0000"
format_stamp: "Formatted at 2026-06-20 11:33:34 on dist-test-slave-ctbw"
server_key: "6e21919af7d093c6f6affd6bd07ea78f"
server_key_iv: "7cdff2767f60f7947662a9f0baef5e30"
server_key_version: "encryptionkey@0"
I20260620 11:33:34.469277 6563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:34.483330 6563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:34.485435 6563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:34.485597 6563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:34.489419 6563 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:45935
I20260620 11:33:34.489442 6630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:45935 every 8 connection(s)
I20260620 11:33:34.489950 6563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:33:34.490629 6631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:34.492727 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 6563
I20260620 11:33:34.492851 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:33:34.493028 3022 external_mini_cluster.cc:1428] Setting key 440bbbb0ddfab9ecdc85d741fa548da5
I20260620 11:33:34.493054 6631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000: Bootstrap starting.
I20260620 11:33:34.493824 6631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:34.494161 6631 log.cc:824] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:34.494962 6631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000: No bootstrap required, opened a new log
I20260620 11:33:34.497289 6631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5de1259f8067482cbcfaa5dd9f2c0000" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 45935 } }
I20260620 11:33:34.497421 6631 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:34.497442 6631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5de1259f8067482cbcfaa5dd9f2c0000, State: Initialized, Role: FOLLOWER
I20260620 11:33:34.497573 6631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5de1259f8067482cbcfaa5dd9f2c0000" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 45935 } }
I20260620 11:33:34.497653 6631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:33:34.497676 6631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:33:34.497766 6631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:34.498301 6631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5de1259f8067482cbcfaa5dd9f2c0000" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 45935 } }
I20260620 11:33:34.498420 6631 leader_election.cc:302] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5de1259f8067482cbcfaa5dd9f2c0000; no voters:
I20260620 11:33:34.498605 6631 leader_election.cc:288] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:33:34.498626 6636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:34.498740 6636 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000 [term 1 LEADER]: Becoming Leader. State: Replica: 5de1259f8067482cbcfaa5dd9f2c0000, State: Running, Role: LEADER
I20260620 11:33:34.498832 6636 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5de1259f8067482cbcfaa5dd9f2c0000" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 45935 } }
I20260620 11:33:34.498915 6631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:34.499187 6638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5de1259f8067482cbcfaa5dd9f2c0000. Latest consensus state: current_term: 1 leader_uuid: "5de1259f8067482cbcfaa5dd9f2c0000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5de1259f8067482cbcfaa5dd9f2c0000" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 45935 } } }
I20260620 11:33:34.499284 6638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000 [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:34.499482 6637 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5de1259f8067482cbcfaa5dd9f2c0000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5de1259f8067482cbcfaa5dd9f2c0000" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 45935 } } }
I20260620 11:33:34.499549 6637 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000 [sys.catalog]: This master's current role is: LEADER
Jun 20 11:33:34 dist-test-slave-ctbw krb5kdc[6550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
W20260620 11:33:34.500759 6652 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:33:34.501398 6652 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:33:34.501520 6641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:33:34.501756 6641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:33:34.503377 6641 catalog_manager.cc:1347] Generated new cluster ID: 418b0cb8ab23450182231e5cae4dfc04
I20260620 11:33:34.503433 6641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:33:34.505776 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:34.493563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55470 (local address 127.2.243.190:45935)
0620 11:33:34.493745 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:34.493754 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:34.493799 (+ 45us) server_negotiation.cc:407] Connection header received
0620 11:33:34.494394 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:34.494409 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:34.494460 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:34.494576 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:34.495435 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:34.496836 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:34.497766 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:34.498265 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:34.498448 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:34.501614 (+ 3166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:34.501618 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:34.501618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:34.501631 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:34.503811 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:34.504229 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:34.504233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:34.504233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:34.504345 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:34.504556 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:34.504557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:34.504557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:34.504868 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:34.505039 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:34.505187 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 11:33:34.505335 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20260620 11:33:34.512554 6641 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:33:34.513118 6641 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:33:34.525033 6641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5de1259f8067482cbcfaa5dd9f2c0000: Generated new TSK 0
I20260620 11:33:34.525270 6641 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
I20260620 11:33:34.553211 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:45935
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:34.639134 6657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:34.639353 6657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:34.639376 6657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:34.640542 6657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:34.640588 6657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:34.640615 6657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:33:34.642311 6657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:45935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:34.642616 6657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:34.642884 6657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:34.647895 6663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:34.647917 6662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:34.648016 6665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:34.648428 6657 server_base.cc:1034] running on GCE node
I20260620 11:33:34.648643 6657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:34.648924 6657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:34.650120 6657 hybrid_clock.cc:648] HybridClock initialized: now 1781955214650103 us; error 47 us; skew 500 ppm
Jun 20 11:33:34 dist-test-slave-ctbw krb5kdc[6550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955214, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:34.653122 6657 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:33:34.653628 6657 webserver.cc:460] Webserver started at http://127.2.243.129:35925/ using document root <none> and password file <none>
I20260620 11:33:34.653889 6657 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:34.653962 6657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:34.654115 6657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:34.655270 6657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "ab58dcbca75d4e2c9abcf3d5bb2c96bb"
format_stamp: "Formatted at 2026-06-20 11:33:34 on dist-test-slave-ctbw"
server_key: "39cd4e95c54db2138bc2d4c9512ac4ac"
server_key_iv: "abd29132d2ca99be9078543c4a0ae4ad"
server_key_version: "encryptionkey@0"
I20260620 11:33:34.655659 6657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "ab58dcbca75d4e2c9abcf3d5bb2c96bb"
format_stamp: "Formatted at 2026-06-20 11:33:34 on dist-test-slave-ctbw"
server_key: "39cd4e95c54db2138bc2d4c9512ac4ac"
server_key_iv: "abd29132d2ca99be9078543c4a0ae4ad"
server_key_version: "encryptionkey@0"
I20260620 11:33:34.657023 6657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:33:34.657785 6672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:34.657977 6657 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:34.658119 6657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "ab58dcbca75d4e2c9abcf3d5bb2c96bb"
format_stamp: "Formatted at 2026-06-20 11:33:34 on dist-test-slave-ctbw"
server_key: "39cd4e95c54db2138bc2d4c9512ac4ac"
server_key_iv: "abd29132d2ca99be9078543c4a0ae4ad"
server_key_version: "encryptionkey@0"
I20260620 11:33:34.658200 6657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:34.679492 6657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:34.681967 6657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:34.682138 6657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:34.682392 6657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:34.682714 6657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:34.682775 6657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:34.682816 6657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:34.682858 6657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:34.689795 6657 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:46483
I20260620 11:33:34.689834 6785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:46483 every 8 connection(s)
I20260620 11:33:34.690194 6657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
Jun 20 11:33:34 dist-test-slave-ctbw krb5kdc[6550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955214, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:34.698932 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 6657
I20260620 11:33:34.699051 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:33:34.699287 3022 external_mini_cluster.cc:1428] Setting key 13e764bfef679839a1e8fee37b00ee86
I20260620 11:33:34.699946 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:34.691085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:37679 (local address 127.2.243.190:45935)
0620 11:33:34.691168 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:34.691170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:34.691378 (+ 208us) server_negotiation.cc:407] Connection header received
0620 11:33:34.692334 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:34.692335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:34.692346 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:34.692403 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:34.693021 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:34.693546 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:34.694544 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:34.694762 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:34.694831 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:34.697112 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:34.697114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:34.697114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:34.697120 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:34.698362 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:34.698818 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:34.698819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:34.698820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:34.698908 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:34.699209 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:34.699210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:34.699211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:34.699346 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:34.699448 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:34.699742 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 11:33:34.699827 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 11:33:34.700222 6788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:34.691048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:45935 (local address 127.2.243.129:37679)
0620 11:33:34.691250 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:34.691289 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:34.692223 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:34.692465 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:34.692469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:34.692735 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:34.692929 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:34.692932 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:34.693631 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:34.693632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:34.694449 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:34.694451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:34.694846 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:34.694846 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:34.695020 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:34.695611 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:34.695620 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:34.697015 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:34.698436 (+ 1421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:34.698442 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:34.698443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:34.698707 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:34.698970 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:34.698971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:34.698971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:34.699127 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:34.699446 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:34.699454 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:34.699605 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 11:33:34.699723 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 11:33:34.699976 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":104,"threads_started":1}
I20260620 11:33:34.700798 6786 heartbeater.cc:347] Connected to a master server at 127.2.243.190:45935
I20260620 11:33:34.700906 6786 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:34.701072 6786 heartbeater.cc:511] Master 127.2.243.190:45935 requested a full tablet report, sending...
I20260620 11:33:34.701592 6595 ts_manager.cc:194] Registered new tserver with Master: ab58dcbca75d4e2c9abcf3d5bb2c96bb (127.2.243.129:46483)
I20260620 11:33:34.702179 6595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:37679
WARNING: no policy specified for kudu/127.2.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
I20260620 11:33:34.725351 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:45935
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:34.809188 6791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:34.809442 6791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:34.809481 6791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:34.810616 6791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:34.810683 6791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:34.810727 6791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:33:34.812219 6791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:45935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:34.812496 6791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:34.812736 6791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:34.817708 6796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:34.817699 6797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:34.817699 6799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:34.817865 6791 server_base.cc:1034] running on GCE node
I20260620 11:33:34.818210 6791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:34.818496 6791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:34.819698 6791 hybrid_clock.cc:648] HybridClock initialized: now 1781955214819697 us; error 50 us; skew 500 ppm
Jun 20 11:33:34 dist-test-slave-ctbw krb5kdc[6550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955214, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:34.822320 6791 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:33:34.822916 6791 webserver.cc:460] Webserver started at http://127.2.243.130:33513/ using document root <none> and password file <none>
I20260620 11:33:34.823206 6791 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:34.823284 6791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:34.823444 6791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:34.824563 6791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "29f6f45a7175458aaaa79b5be7a22279"
format_stamp: "Formatted at 2026-06-20 11:33:34 on dist-test-slave-ctbw"
server_key: "9eb1d84a7bb2e18ed1f40d69ece098fe"
server_key_iv: "0e67297bd8093f311400d8ef0706c434"
server_key_version: "encryptionkey@0"
I20260620 11:33:34.824954 6791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "29f6f45a7175458aaaa79b5be7a22279"
format_stamp: "Formatted at 2026-06-20 11:33:34 on dist-test-slave-ctbw"
server_key: "9eb1d84a7bb2e18ed1f40d69ece098fe"
server_key_iv: "0e67297bd8093f311400d8ef0706c434"
server_key_version: "encryptionkey@0"
I20260620 11:33:34.826364 6791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:33:34.827258 6806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:34.827517 6791 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:34.827633 6791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "29f6f45a7175458aaaa79b5be7a22279"
format_stamp: "Formatted at 2026-06-20 11:33:34 on dist-test-slave-ctbw"
server_key: "9eb1d84a7bb2e18ed1f40d69ece098fe"
server_key_iv: "0e67297bd8093f311400d8ef0706c434"
server_key_version: "encryptionkey@0"
I20260620 11:33:34.827739 6791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:34.849629 6791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:34.851948 6791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:34.852113 6791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:34.852363 6791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:34.852692 6791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:34.852747 6791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:34.852798 6791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:34.852839 6791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:34.858851 6791 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:34415
I20260620 11:33:34.858961 6919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:34415 every 8 connection(s)
I20260620 11:33:34.859330 6791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:33:34.860422 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 6791
I20260620 11:33:34.860548 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:33:34.860741 3022 external_mini_cluster.cc:1428] Setting key b49bf2605198cba4fbde2743c6cab2d4
Jun 20 11:33:34 dist-test-slave-ctbw krb5kdc[6550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955214, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.243.131@KRBTEST.COM; defaulting to no policy
I20260620 11:33:34.870069 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:34.860283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:59195 (local address 127.2.243.190:45935)
0620 11:33:34.860357 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:34.860360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:34.860572 (+ 212us) server_negotiation.cc:407] Connection header received
0620 11:33:34.861760 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:34.861761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:34.861772 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:34.861834 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:34.862575 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:34.863171 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:34.864189 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:34.864404 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:34.864491 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:34.867111 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:34.867113 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:34.867113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:34.867121 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:34.868569 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:34.869022 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:34.869024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:34.869024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:34.869086 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:34.869356 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:34.869357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:34.869357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:34.869485 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:34.869565 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:34.869875 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 11:33:34.869964 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 11:33:34.870266 6922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:34.860180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:45935 (local address 127.2.243.130:59195)
0620 11:33:34.860445 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:34.860495 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:34.861659 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:34.861913 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:34.861917 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:34.862234 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:34.862488 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:34.862492 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:34.863262 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:34.863263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:34.864088 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:34.864090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:34.864499 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:34.864500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:34.864640 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:34.865235 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:34.865244 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:34.867002 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:34.868672 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:34.868678 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:34.868681 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:34.868943 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:34.869156 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:34.869157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:34.869158 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:34.869269 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:34.869556 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:34.869562 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:34.869739 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 11:33:34.869854 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 11:33:34.870026 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":114,"threads_started":1}
I20260620 11:33:34.870921 6920 heartbeater.cc:347] Connected to a master server at 127.2.243.190:45935
I20260620 11:33:34.871033 6920 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:34.871201 6920 heartbeater.cc:511] Master 127.2.243.190:45935 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.131@KRBTEST.COM" created.
I20260620 11:33:34.871630 6595 ts_manager.cc:194] Registered new tserver with Master: 29f6f45a7175458aaaa79b5be7a22279 (127.2.243.130:34415)
I20260620 11:33:34.872007 6595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:59195
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
I20260620 11:33:34.886166 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:45935
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:34.968456 6925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:34.968670 6925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:34.968695 6925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:34.969765 6925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:34.969802 6925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:34.969830 6925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:33:34.971365 6925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:45935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:34.971555 6925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:34.971843 6925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:34.976967 6933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:34.976984 6930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:34.976981 6931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:34.977190 6925 server_base.cc:1034] running on GCE node
I20260620 11:33:34.977391 6925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:34.977638 6925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:34.978808 6925 hybrid_clock.cc:648] HybridClock initialized: now 1781955214978792 us; error 57 us; skew 500 ppm
Jun 20 11:33:34 dist-test-slave-ctbw krb5kdc[6550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955214, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:34.981359 6925 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:33:34.981781 6925 webserver.cc:460] Webserver started at http://127.2.243.131:38883/ using document root <none> and password file <none>
I20260620 11:33:34.982007 6925 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:34.982072 6925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:34.982223 6925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:34.983470 6925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "d7b250ff66194bc993ac79238e97ba8f"
format_stamp: "Formatted at 2026-06-20 11:33:34 on dist-test-slave-ctbw"
server_key: "ab2c234fef8715755c710a9f99d17737"
server_key_iv: "769ae55ef67096d6e8a3a15b7a31acbe"
server_key_version: "encryptionkey@0"
I20260620 11:33:34.983848 6925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "d7b250ff66194bc993ac79238e97ba8f"
format_stamp: "Formatted at 2026-06-20 11:33:34 on dist-test-slave-ctbw"
server_key: "ab2c234fef8715755c710a9f99d17737"
server_key_iv: "769ae55ef67096d6e8a3a15b7a31acbe"
server_key_version: "encryptionkey@0"
I20260620 11:33:34.985194 6925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:33:34.985999 6940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:34.986198 6925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:34.986295 6925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "d7b250ff66194bc993ac79238e97ba8f"
format_stamp: "Formatted at 2026-06-20 11:33:34 on dist-test-slave-ctbw"
server_key: "ab2c234fef8715755c710a9f99d17737"
server_key_iv: "769ae55ef67096d6e8a3a15b7a31acbe"
server_key_version: "encryptionkey@0"
I20260620 11:33:34.986370 6925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:34.998353 6925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:35.001219 6925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:35.001385 6925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:35.001632 6925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:35.001943 6925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:35.001996 6925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:35.002040 6925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:35.002065 6925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:35.008404 6925 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:32943
I20260620 11:33:35.008455 7053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:32943 every 8 connection(s)
I20260620 11:33:35.008805 6925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:33:35.010982 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 6925
I20260620 11:33:35.011075 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:33:35.011245 3022 external_mini_cluster.cc:1428] Setting key 81060965c5ad3f5f765b20b5b3fb5d1d
Jun 20 11:33:35 dist-test-slave-ctbw krb5kdc[6550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955214, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:35.019068 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:35.009669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:56103 (local address 127.2.243.190:45935)
0620 11:33:35.009786 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:35.009788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:35.009961 (+ 173us) server_negotiation.cc:407] Connection header received
0620 11:33:35.011057 (+ 1096us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:35.011059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:35.011070 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:35.011125 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:35.011933 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:35.012435 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:35.013444 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:35.013644 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:35.013727 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:35.015988 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:35.015989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:35.015990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:35.015997 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:35.017610 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:35.018067 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:35.018068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:35.018069 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:35.018130 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:35.018373 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:35.018374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:35.018375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:35.018499 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:35.018572 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:35.018857 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 11:33:35.018977 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 11:33:35.019251 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:35.009598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:45935 (local address 127.2.243.131:56103)
0620 11:33:35.009827 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:35.009873 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:35.010951 (+ 1078us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:35.011200 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:35.011204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:35.011566 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:35.011839 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:35.011844 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:35.012525 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:35.012526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:35.013344 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:35.013346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:35.013735 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:35.013736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:35.013878 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:35.014434 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:35.014442 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:35.015854 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:35.017725 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:35.017732 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:35.017734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:35.017967 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:35.018198 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:35.018199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:35.018199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:35.018304 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:35.018574 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:35.018581 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:35.018717 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 11:33:35.018830 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 11:33:35.019012 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":89,"threads_started":1}
I20260620 11:33:35.019797 7054 heartbeater.cc:347] Connected to a master server at 127.2.243.190:45935
I20260620 11:33:35.019905 7054 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:35.020045 7054 heartbeater.cc:511] Master 127.2.243.190:45935 requested a full tablet report, sending...
I20260620 11:33:35.020381 6595 ts_manager.cc:194] Registered new tserver with Master: d7b250ff66194bc993ac79238e97ba8f (127.2.243.131:32943)
I20260620 11:33:35.020736 6595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:56103
I20260620 11:33:35.023149 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 11:33:35 dist-test-slave-ctbw krb5kdc[6550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 11:33:35 dist-test-slave-ctbw krb5kdc[6550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.129@KRBTEST.COM
I20260620 11:33:35.046267 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:35.038438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44218 (local address 127.2.243.129:46483)
0620 11:33:35.038636 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:35.038641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:35.038652 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:33:35.038677 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:35.038680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:35.038693 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:35.038761 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:35.039355 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:35.040084 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:35.040788 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:35.041121 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:35.041176 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:35.043544 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:35.043557 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:35.043557 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:35.043569 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:35.045061 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:35.045412 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:35.045416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:35.045416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:35.045491 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:35.045693 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:35.045694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:35.045694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:35.045832 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:35.046015 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:35.046060 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 11:33:35.046139 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":110,"threads_started":1}
Jun 20 11:33:35 dist-test-slave-ctbw krb5kdc[6550](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:35.054395 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:35.047803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55486 (local address 127.2.243.190:45935)
0620 11:33:35.047862 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:35.047864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:35.047872 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:33:35.047952 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:35.047953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:35.047962 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:35.048008 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:35.048571 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:35.049043 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:35.049712 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:35.049893 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:35.049960 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:35.052083 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:35.052085 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:35.052085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:35.052091 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:35.053306 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:35.053653 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:35.053654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:35.053655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:35.053705 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:35.053876 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:35.053876 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:35.053877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:35.054021 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:35.054112 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:35.054273 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 11:33:35.054331 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 11:33:35.054651 6595 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55486
I20260620 11:33:35.055039 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 6657
I20260620 11:33:35.059444 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 6791
I20260620 11:33:35.063297 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 6925
I20260620 11:33:35.067386 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 6563
2026-06-20T11:33:35Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1871 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:33:35 dist-test-slave-ctbw krb5kdc[7066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:33:35 dist-test-slave-ctbw krb5kdc[7066](info): set up 2 sockets
Jun 20 11:33:35 dist-test-slave-ctbw krb5kdc[7066](info): commencing operation
krb5kdc: starting...
W20260620 11:33:36.115161 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:33:36 dist-test-slave-ctbw krb5kdc[7066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:33:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:33:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:33:36.233846 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:36343
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:46651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:36343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:36.315855 7079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:36.316110 7079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:36.316149 7079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:36.317320 7079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:33:36.317394 7079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:36.317416 7079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:36.317445 7079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:33:36.317507 7079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:33:36.319077 7079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:46651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:36343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:36343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:36.319360 7079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:36.319594 7079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:36.324812 7087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:36.324826 7084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:36.324826 7085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:36.325116 7079 server_base.cc:1034] running on GCE node
I20260620 11:33:36.325470 7079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:36.325729 7079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:36.326941 7079 hybrid_clock.cc:648] HybridClock initialized: now 1781955216326919 us; error 43 us; skew 500 ppm
Jun 20 11:33:36 dist-test-slave-ctbw krb5kdc[7066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955216, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:36.332528 7079 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:33:36.332897 7079 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:36.332968 7079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:36.333114 7079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:36.334198 7079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "c135676571fb426da40c3425a694cfcb"
format_stamp: "Formatted at 2026-06-20 11:33:36 on dist-test-slave-ctbw"
server_key: "14f93916243c18aa746c360bb8ba3bde"
server_key_iv: "d3f4a992565ecd910ef3371934585025"
server_key_version: "encryptionkey@0"
I20260620 11:33:36.334600 7079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "c135676571fb426da40c3425a694cfcb"
format_stamp: "Formatted at 2026-06-20 11:33:36 on dist-test-slave-ctbw"
server_key: "14f93916243c18aa746c360bb8ba3bde"
server_key_iv: "d3f4a992565ecd910ef3371934585025"
server_key_version: "encryptionkey@0"
I20260620 11:33:36.335953 7079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:33:36.336767 7093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:36.336982 7079 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:36.337100 7079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "c135676571fb426da40c3425a694cfcb"
format_stamp: "Formatted at 2026-06-20 11:33:36 on dist-test-slave-ctbw"
server_key: "14f93916243c18aa746c360bb8ba3bde"
server_key_iv: "d3f4a992565ecd910ef3371934585025"
server_key_version: "encryptionkey@0"
I20260620 11:33:36.337172 7079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:36.357125 7079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:36.359601 7079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:36.359781 7079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:36.363811 7079 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:36343
I20260620 11:33:36.363848 7145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:36343 every 8 connection(s)
I20260620 11:33:36.364168 7079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:33:36.365142 7146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:36.367697 7146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb: Bootstrap starting.
I20260620 11:33:36.368335 7146 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:36.368664 7146 log.cc:824] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:36.369051 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 7079
I20260620 11:33:36.369180 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:33:36.369398 3022 external_mini_cluster.cc:1428] Setting key 3ed3133c0e1632805e461c21929011f4
I20260620 11:33:36.369586 7146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb: No bootstrap required, opened a new log
I20260620 11:33:36.371543 7146 raft_consensus.cc:348] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c135676571fb426da40c3425a694cfcb" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36343 } }
I20260620 11:33:36.371716 7146 raft_consensus.cc:374] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:36.371763 7146 raft_consensus.cc:729] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c135676571fb426da40c3425a694cfcb, State: Initialized, Role: FOLLOWER
I20260620 11:33:36.371923 7146 consensus_queue.cc:260] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c135676571fb426da40c3425a694cfcb" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36343 } }
I20260620 11:33:36.372035 7146 raft_consensus.cc:388] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:33:36.372084 7146 raft_consensus.cc:482] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:33:36.372144 7146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:36.372993 7146 raft_consensus.cc:504] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c135676571fb426da40c3425a694cfcb" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36343 } }
I20260620 11:33:36.373116 7146 leader_election.cc:302] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c135676571fb426da40c3425a694cfcb; no voters:
I20260620 11:33:36.373332 7146 leader_election.cc:288] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:33:36.373370 7151 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:36.373492 7151 raft_consensus.cc:686] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb [term 1 LEADER]: Becoming Leader. State: Replica: c135676571fb426da40c3425a694cfcb, State: Running, Role: LEADER
I20260620 11:33:36.373577 7151 consensus_queue.cc:237] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c135676571fb426da40c3425a694cfcb" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36343 } }
I20260620 11:33:36.373610 7146 sys_catalog.cc:564] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:36.373911 7153 sys_catalog.cc:455] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c135676571fb426da40c3425a694cfcb. Latest consensus state: current_term: 1 leader_uuid: "c135676571fb426da40c3425a694cfcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c135676571fb426da40c3425a694cfcb" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36343 } } }
I20260620 11:33:36.374022 7153 sys_catalog.cc:458] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:36.374266 7155 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:33:36.374853 7155 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:33:36.375218 7152 sys_catalog.cc:455] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c135676571fb426da40c3425a694cfcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c135676571fb426da40c3425a694cfcb" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36343 } } }
I20260620 11:33:36.375303 7152 sys_catalog.cc:458] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:36.376765 7155 catalog_manager.cc:1347] Generated new cluster ID: 160ad6e873ea41e2a58f134cad30ebeb
I20260620 11:33:36.376825 7155 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:33:36 dist-test-slave-ctbw krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:36.383075 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:36.369998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51792 (local address 127.2.243.190:36343)
0620 11:33:36.370212 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:36.370220 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:36.370247 (+ 27us) server_negotiation.cc:407] Connection header received
0620 11:33:36.371525 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:36.371537 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:36.371593 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:36.371744 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:36.372486 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.373608 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.374791 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.375588 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.375730 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:36.378733 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:36.378737 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:36.378738 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:36.378752 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:36.380906 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:36.381329 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:36.381333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:36.381333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:36.381457 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:36.381664 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:36.381665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:36.381665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:36.381985 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:36.382150 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:36.382649 (+ 499us) server_negotiation.cc:299] Negotiation successful
0620 11:33:36.382824 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":84,"threads_started":1}
I20260620 11:33:36.395704 7155 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:33:36.396463 7155 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:33:36.411630 7155 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb: Generated new TSK 0
I20260620 11:33:36.411881 7155 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
I20260620 11:33:36.441663 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:36343
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:46651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:36.524333 7172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:36.524547 7172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:36.524575 7172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:36.525753 7172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:36.525801 7172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:36.525831 7172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:33:36.527442 7172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:46651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:36343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:36.527698 7172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:36.527962 7172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:36.533030 7180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:36.533039 7177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:36.533254 7172 server_base.cc:1034] running on GCE node
W20260620 11:33:36.533046 7178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:36.533612 7172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:36.533866 7172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:36.535048 7172 hybrid_clock.cc:648] HybridClock initialized: now 1781955216535033 us; error 32 us; skew 500 ppm
Jun 20 11:33:36 dist-test-slave-ctbw krb5kdc[7066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955216, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:36.537616 7172 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:33:36.537966 7172 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:36.538057 7172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:36.538213 7172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:36.539650 7172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "7e5c008b2af24a81bf40f01ee8e4c08f"
format_stamp: "Formatted at 2026-06-20 11:33:36 on dist-test-slave-ctbw"
server_key: "b4aa2df7412a1ef6ad8874c719b63df6"
server_key_iv: "8c6f41ef6b04ffa558e963a118c32746"
server_key_version: "encryptionkey@0"
I20260620 11:33:36.540033 7172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "7e5c008b2af24a81bf40f01ee8e4c08f"
format_stamp: "Formatted at 2026-06-20 11:33:36 on dist-test-slave-ctbw"
server_key: "b4aa2df7412a1ef6ad8874c719b63df6"
server_key_iv: "8c6f41ef6b04ffa558e963a118c32746"
server_key_version: "encryptionkey@0"
I20260620 11:33:36.541466 7172 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 11:33:36.542210 7186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:36.542385 7172 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:33:36.542470 7172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "7e5c008b2af24a81bf40f01ee8e4c08f"
format_stamp: "Formatted at 2026-06-20 11:33:36 on dist-test-slave-ctbw"
server_key: "b4aa2df7412a1ef6ad8874c719b63df6"
server_key_iv: "8c6f41ef6b04ffa558e963a118c32746"
server_key_version: "encryptionkey@0"
I20260620 11:33:36.542577 7172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:36.556355 7172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:36.558679 7172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:36.558842 7172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:36.559124 7172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:36.559428 7172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:36.559460 7172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:36.559517 7172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:36.559551 7172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:36.565280 7172 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:42095
I20260620 11:33:36.565320 7299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:42095 every 8 connection(s)
I20260620 11:33:36.565598 7172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:33:36.566833 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 7172
I20260620 11:33:36.566972 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:33:36.567184 3022 external_mini_cluster.cc:1428] Setting key 9e8007dd6b0034dc87a25eed339c17dc
Jun 20 11:33:36 dist-test-slave-ctbw krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955216, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.243.130@KRBTEST.COM; defaulting to no policy
I20260620 11:33:36.575418 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:36.566242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:48013 (local address 127.2.243.190:36343)
0620 11:33:36.566321 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:36.566324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:36.566511 (+ 187us) server_negotiation.cc:407] Connection header received
0620 11:33:36.567099 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:36.567101 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:36.567114 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:36.567172 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:36.568279 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.568779 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.569791 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.570014 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.570098 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:36.572337 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:36.572338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:36.572339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:36.572348 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:36.573845 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:36.574383 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:36.574385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:36.574386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:36.574460 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:36.574727 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:36.574728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:36.574729 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:36.574863 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:36.574948 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:36.575281 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 11:33:36.575350 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 11:33:36.575712 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:36.566209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:36343 (local address 127.2.243.129:48013)
0620 11:33:36.566386 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:36.566422 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:36.566944 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:36.567249 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:36.567269 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:36.567618 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:36.568181 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:36.568186 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.568878 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.568879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:36.569694 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:36.569695 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.570106 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.570106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:36.570225 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:36.570760 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:36.570769 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:36.572225 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:36.573945 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:36.573951 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:36.573953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:36.574284 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:36.574526 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:36.574527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:36.574527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:36.574648 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:36.574946 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:36.574953 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:36.575132 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 11:33:36.575265 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 11:33:36.575399 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":104,"threads_started":1}
I20260620 11:33:36.576287 7300 heartbeater.cc:347] Connected to a master server at 127.2.243.190:36343
I20260620 11:33:36.576381 7300 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:36.576510 7300 heartbeater.cc:511] Master 127.2.243.190:36343 requested a full tablet report, sending...
I20260620 11:33:36.577061 7110 ts_manager.cc:194] Registered new tserver with Master: 7e5c008b2af24a81bf40f01ee8e4c08f (127.2.243.129:42095)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.130@KRBTEST.COM" created.
I20260620 11:33:36.577947 7110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:48013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
I20260620 11:33:36.593533 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:36343
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:46651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:36.675325 7305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:36.675630 7305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:36.675678 7305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:36.676901 7305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:36.676976 7305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:36.677026 7305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:33:36.678586 7305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:46651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:36343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:36.678861 7305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:36.679148 7305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:36.684203 7313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:36.684211 7311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:36.684207 7310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:36.684543 7305 server_base.cc:1034] running on GCE node
I20260620 11:33:36.684775 7305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:36.684973 7305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:36.686139 7305 hybrid_clock.cc:648] HybridClock initialized: now 1781955216686121 us; error 39 us; skew 500 ppm
Jun 20 11:33:36 dist-test-slave-ctbw krb5kdc[7066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955216, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:36.688715 7305 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:33:36.689087 7305 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:36.689139 7305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:36.689271 7305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:36.690410 7305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "39ab21cc03214852bd69974973b92cac"
format_stamp: "Formatted at 2026-06-20 11:33:36 on dist-test-slave-ctbw"
server_key: "fe159eb6f9962dfb45b5b35dc62ee14b"
server_key_iv: "ea59940621b65ca3ece2374e33a8a279"
server_key_version: "encryptionkey@0"
I20260620 11:33:36.690776 7305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "39ab21cc03214852bd69974973b92cac"
format_stamp: "Formatted at 2026-06-20 11:33:36 on dist-test-slave-ctbw"
server_key: "fe159eb6f9962dfb45b5b35dc62ee14b"
server_key_iv: "ea59940621b65ca3ece2374e33a8a279"
server_key_version: "encryptionkey@0"
I20260620 11:33:36.692157 7305 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:36.692896 7319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:36.693127 7305 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:36.693203 7305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "39ab21cc03214852bd69974973b92cac"
format_stamp: "Formatted at 2026-06-20 11:33:36 on dist-test-slave-ctbw"
server_key: "fe159eb6f9962dfb45b5b35dc62ee14b"
server_key_iv: "ea59940621b65ca3ece2374e33a8a279"
server_key_version: "encryptionkey@0"
I20260620 11:33:36.693289 7305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:36.710503 7305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:36.712522 7305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:36.712685 7305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:36.712935 7305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:36.713265 7305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:36.713299 7305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:36.713320 7305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:36.713366 7305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:36.719573 7305 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:35517
I20260620 11:33:36.719660 7432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:35517 every 8 connection(s)
I20260620 11:33:36.719901 7305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
Jun 20 11:33:36 dist-test-slave-ctbw krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955216, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:36.728574 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 7305
I20260620 11:33:36.728696 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:33:36.728912 3022 external_mini_cluster.cc:1428] Setting key d43fb49cd3bc07d16f9f9977ec04cb61
I20260620 11:33:36.730149 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:36.720750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:39109 (local address 127.2.243.190:36343)
0620 11:33:36.720826 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:36.720829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:36.720986 (+ 157us) server_negotiation.cc:407] Connection header received
0620 11:33:36.721614 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:36.721615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:36.721631 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:36.721694 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:36.722660 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.723249 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.724286 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.724479 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.724592 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:36.726815 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:36.726817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:36.726818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:36.726825 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:36.728359 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:36.728883 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:36.728884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:36.728885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:36.728949 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:36.729322 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:36.729324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:36.729324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:36.729472 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:36.729537 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:36.729917 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 11:33:36.730027 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 11:33:36.730213 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:36.720611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:36343 (local address 127.2.243.130:39109)
0620 11:33:36.720841 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:36.720895 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:36.721490 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:36.721768 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:36.721772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:36.722054 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:36.722527 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:36.722532 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.723340 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.723341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:36.724176 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:36.724178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.724600 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.724602 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:36.724732 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:36.725282 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:36.725292 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:36.726716 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:36.728475 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:36.728481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:36.728483 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:36.728805 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:36.729043 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:36.729043 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:36.729044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:36.729218 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:36.729545 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:36.729550 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:36.729720 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 11:33:36.729850 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 11:33:36.730014 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":89,"threads_started":1}
I20260620 11:33:36.730723 7433 heartbeater.cc:347] Connected to a master server at 127.2.243.190:36343
I20260620 11:33:36.730830 7433 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:36.731010 7433 heartbeater.cc:511] Master 127.2.243.190:36343 requested a full tablet report, sending...
I20260620 11:33:36.731546 7110 ts_manager.cc:194] Registered new tserver with Master: 39ab21cc03214852bd69974973b92cac (127.2.243.130:35517)
I20260620 11:33:36.731945 7110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:39109
WARNING: no policy specified for kudu/127.2.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
I20260620 11:33:36.757308 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:36343
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:46651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:36.837996 7438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:36.838241 7438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:36.838282 7438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:36.839563 7438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:36.839622 7438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:36.839668 7438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:33:36.841300 7438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:46651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:36343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:36.841563 7438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:36.841789 7438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:36.847061 7446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:36.847071 7444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:36.847082 7438 server_base.cc:1034] running on GCE node
W20260620 11:33:36.847064 7443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:36.847430 7438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:36.847715 7438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:36.848878 7438 hybrid_clock.cc:648] HybridClock initialized: now 1781955216848866 us; error 45 us; skew 500 ppm
Jun 20 11:33:36 dist-test-slave-ctbw krb5kdc[7066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955216, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:36.851503 7438 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:33:36.851907 7438 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:36.851986 7438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:36.852137 7438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:36.853191 7438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "becdf81f1c2b4d51a2d39a7c352077e4"
format_stamp: "Formatted at 2026-06-20 11:33:36 on dist-test-slave-ctbw"
server_key: "c7e8afcf1d4f6d50f1f1da3d3a90d7af"
server_key_iv: "ab69af98d1594c498708281213a97dc4"
server_key_version: "encryptionkey@0"
I20260620 11:33:36.853545 7438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "becdf81f1c2b4d51a2d39a7c352077e4"
format_stamp: "Formatted at 2026-06-20 11:33:36 on dist-test-slave-ctbw"
server_key: "c7e8afcf1d4f6d50f1f1da3d3a90d7af"
server_key_iv: "ab69af98d1594c498708281213a97dc4"
server_key_version: "encryptionkey@0"
I20260620 11:33:36.854854 7438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:33:36.855608 7452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:36.855804 7438 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:33:36.855901 7438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "becdf81f1c2b4d51a2d39a7c352077e4"
format_stamp: "Formatted at 2026-06-20 11:33:36 on dist-test-slave-ctbw"
server_key: "c7e8afcf1d4f6d50f1f1da3d3a90d7af"
server_key_iv: "ab69af98d1594c498708281213a97dc4"
server_key_version: "encryptionkey@0"
I20260620 11:33:36.855983 7438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:36.870136 7438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:36.872587 7438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:36.872750 7438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:36.873034 7438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:36.873378 7438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:36.873435 7438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:36.873479 7438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:36.873521 7438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:36.879668 7438 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:39051
I20260620 11:33:36.879765 7565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:39051 every 8 connection(s)
I20260620 11:33:36.880010 7438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:33:36.882370 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 7438
I20260620 11:33:36.882481 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:33:36.882712 3022 external_mini_cluster.cc:1428] Setting key edc285e53765477adbdbf01710bafd85
Jun 20 11:33:36 dist-test-slave-ctbw krb5kdc[7066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955216, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:36.889756 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:36.880728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:38917 (local address 127.2.243.190:36343)
0620 11:33:36.880780 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:36.880782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:36.880956 (+ 174us) server_negotiation.cc:407] Connection header received
0620 11:33:36.881442 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:36.881443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:36.881452 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:36.881517 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:36.882695 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.883305 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.884357 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.884591 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.884652 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:36.886745 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:36.886746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:36.886746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:36.886753 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:36.888279 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:36.888752 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:36.888753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:36.888753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:36.888810 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:36.889116 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:36.889117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:36.889118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:36.889296 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:36.889353 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:36.889619 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 11:33:36.889684 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:33:36.889948 7568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:36.880690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:36343 (local address 127.2.243.131:38917)
0620 11:33:36.880840 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:36.880882 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:36.881355 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:36.881590 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:36.881593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:36.881836 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:36.882531 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:36.882539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.883401 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.883402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:36.884266 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:36.884269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.884667 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.884667 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:36.884764 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:36.885279 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:36.885288 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:36.886658 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:36.888354 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:36.888359 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:36.888361 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:36.888576 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:36.888878 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:36.888878 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:36.888879 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:36.888980 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:36.889358 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:36.889362 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:36.889483 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 11:33:36.889581 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 11:33:36.889702 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260620 11:33:36.890473 7566 heartbeater.cc:347] Connected to a master server at 127.2.243.190:36343
I20260620 11:33:36.890599 7566 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:36.890769 7566 heartbeater.cc:511] Master 127.2.243.190:36343 requested a full tablet report, sending...
I20260620 11:33:36.891180 7110 ts_manager.cc:194] Registered new tserver with Master: becdf81f1c2b4d51a2d39a7c352077e4 (127.2.243.131:39051)
I20260620 11:33:36.891522 7110 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:38917
I20260620 11:33:36.894351 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:33:36.901999 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:36.895720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51800 (local address 127.2.243.190:36343)
0620 11:33:36.895780 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:36.895782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:36.895822 (+ 40us) server_negotiation.cc:407] Connection header received
0620 11:33:36.895898 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:36.895898 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:36.895908 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:36.895980 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:36.896561 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.897037 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.897826 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.898013 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.898096 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:36.899394 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:36.899395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:36.899396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:36.899402 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:36.900845 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:36.901198 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:36.901199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:36.901200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:36.901261 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:36.901459 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:36.901460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:36.901461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:36.901606 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:36.901707 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:36.901819 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 11:33:36.901897 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:33:36.903652 7110 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51800:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:33:36.905143 7110 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:33:36.911235 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:36.907765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41006 (local address 127.2.243.129:42095)
0620 11:33:36.907908 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:36.907912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:36.907920 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:33:36.907955 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:36.907959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:36.907976 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:36.908258 (+ 282us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:36.908335 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.909006 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.910145 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.910770 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.910845 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:36.910922 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:36.911111 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 11:33:36.911152 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260620 11:33:36.911224 7574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:36.907515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:42095 (local address 127.0.0.1:41006)
0620 11:33:36.907676 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:36.907686 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:36.907739 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:36.908120 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:36.908121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:36.908123 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:36.908179 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:36.908180 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.909109 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.909110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:36.910046 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:36.910050 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.910905 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.910907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:36.910993 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:36.911009 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 11:33:36.911088 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 11:33:36.911142 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":35,"thread_start_us":104,"threads_started":1}
I20260620 11:33:36.911621 7578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:36.907308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:39051 (local address 127.0.0.1:36068)
0620 11:33:36.907679 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:36.907690 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:36.907734 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:36.908187 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:36.908188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:36.908191 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:36.908227 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:36.908229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.909431 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.909432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:36.910499 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:36.910501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.911398 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.911398 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:36.911458 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:36.911462 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:33:36.911519 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 11:33:36.911558 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":64,"threads_started":1}
I20260620 11:33:36.911641 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:36.907383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36068 (local address 127.2.243.131:39051)
0620 11:33:36.907961 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:36.907966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:36.907975 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:33:36.908010 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:36.908012 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:36.908026 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:36.908100 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:36.908306 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.909332 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.910600 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.911320 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.911395 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:36.911448 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:36.911532 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 11:33:36.911568 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":74,"threads_started":1}
I20260620 11:33:36.911779 7577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:36.907424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60214 (local address 127.2.243.130:35517)
0620 11:33:36.907605 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:36.907610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:36.907622 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:33:36.907727 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:36.907731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:36.907759 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:36.907829 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:36.908019 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.909173 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.910595 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.911327 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.911404 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:36.911459 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:36.911642 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 11:33:36.911683 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":76,"threads_started":1}
I20260620 11:33:36.912022 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:36.907307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:35517 (local address 127.0.0.1:60214)
0620 11:33:36.907480 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:36.907519 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:36.907587 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:36.907842 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:36.907845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:36.907847 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:36.907919 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:36.907921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.909295 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.909295 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:36.910505 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:36.910507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.911495 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.911496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:36.911556 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:36.911560 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:33:36.911916 (+ 356us) client_negotiation.cc:240] Negotiation successful
0620 11:33:36.911958 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":34,"thread_start_us":76,"threads_started":1}
I20260620 11:33:36.912367 7500 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0ed0f46bc74ed7a83a9b77f0f99d9c (DEFAULT_TABLE table=test-table [id=25ed378080e94eb29392ac897d3dd5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:36.912735 7500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0ed0f46bc74ed7a83a9b77f0f99d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:36.912711 7234 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0ed0f46bc74ed7a83a9b77f0f99d9c (DEFAULT_TABLE table=test-table [id=25ed378080e94eb29392ac897d3dd5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:36.913039 7234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0ed0f46bc74ed7a83a9b77f0f99d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:36.915355 7367 tablet_service.cc:1467] Processing CreateTablet for tablet 2e0ed0f46bc74ed7a83a9b77f0f99d9c (DEFAULT_TABLE table=test-table [id=25ed378080e94eb29392ac897d3dd5d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:36.915712 7367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e0ed0f46bc74ed7a83a9b77f0f99d9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:36.915736 7581 tablet_bootstrap.cc:492] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f: Bootstrap starting.
I20260620 11:33:36.916255 7581 tablet_bootstrap.cc:654] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:36.916783 7581 log.cc:824] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:36.917114 7580 tablet_bootstrap.cc:492] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P becdf81f1c2b4d51a2d39a7c352077e4: Bootstrap starting.
I20260620 11:33:36.917520 7581 tablet_bootstrap.cc:492] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f: No bootstrap required, opened a new log
I20260620 11:33:36.917589 7581 ts_tablet_manager.cc:1397] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:33:36.917739 7580 tablet_bootstrap.cc:654] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P becdf81f1c2b4d51a2d39a7c352077e4: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:36.918327 7580 log.cc:824] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P becdf81f1c2b4d51a2d39a7c352077e4: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:36.918378 7583 tablet_bootstrap.cc:492] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 39ab21cc03214852bd69974973b92cac: Bootstrap starting.
I20260620 11:33:36.919080 7583 tablet_bootstrap.cc:654] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 39ab21cc03214852bd69974973b92cac: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:36.919114 7580 tablet_bootstrap.cc:492] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P becdf81f1c2b4d51a2d39a7c352077e4: No bootstrap required, opened a new log
I20260620 11:33:36.919169 7580 ts_tablet_manager.cc:1397] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P becdf81f1c2b4d51a2d39a7c352077e4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:33:36.919695 7583 log.cc:824] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 39ab21cc03214852bd69974973b92cac: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:36.919785 7581 raft_consensus.cc:348] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "becdf81f1c2b4d51a2d39a7c352077e4" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 39051 } } peers { permanent_uuid: "39ab21cc03214852bd69974973b92cac" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 35517 } } peers { permanent_uuid: "7e5c008b2af24a81bf40f01ee8e4c08f" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42095 } }
I20260620 11:33:36.919987 7581 raft_consensus.cc:374] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:36.920029 7581 raft_consensus.cc:729] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e5c008b2af24a81bf40f01ee8e4c08f, State: Initialized, Role: FOLLOWER
I20260620 11:33:36.920218 7581 consensus_queue.cc:260] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "becdf81f1c2b4d51a2d39a7c352077e4" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 39051 } } peers { permanent_uuid: "39ab21cc03214852bd69974973b92cac" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 35517 } } peers { permanent_uuid: "7e5c008b2af24a81bf40f01ee8e4c08f" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42095 } }
I20260620 11:33:36.920436 7583 tablet_bootstrap.cc:492] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 39ab21cc03214852bd69974973b92cac: No bootstrap required, opened a new log
I20260620 11:33:36.920490 7581 ts_tablet_manager.cc:1428] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:33:36.920503 7583 ts_tablet_manager.cc:1397] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 39ab21cc03214852bd69974973b92cac: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:33:36.920614 7300 heartbeater.cc:503] Master 127.2.243.190:36343 was elected leader, sending a full tablet report...
I20260620 11:33:36.921381 7580 raft_consensus.cc:348] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P becdf81f1c2b4d51a2d39a7c352077e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "becdf81f1c2b4d51a2d39a7c352077e4" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 39051 } } peers { permanent_uuid: "39ab21cc03214852bd69974973b92cac" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 35517 } } peers { permanent_uuid: "7e5c008b2af24a81bf40f01ee8e4c08f" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42095 } }
I20260620 11:33:36.921543 7580 raft_consensus.cc:374] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P becdf81f1c2b4d51a2d39a7c352077e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:36.921578 7580 raft_consensus.cc:729] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P becdf81f1c2b4d51a2d39a7c352077e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: becdf81f1c2b4d51a2d39a7c352077e4, State: Initialized, Role: FOLLOWER
I20260620 11:33:36.921684 7580 consensus_queue.cc:260] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P becdf81f1c2b4d51a2d39a7c352077e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "becdf81f1c2b4d51a2d39a7c352077e4" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 39051 } } peers { permanent_uuid: "39ab21cc03214852bd69974973b92cac" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 35517 } } peers { permanent_uuid: "7e5c008b2af24a81bf40f01ee8e4c08f" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42095 } }
I20260620 11:33:36.921952 7580 ts_tablet_manager.cc:1428] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P becdf81f1c2b4d51a2d39a7c352077e4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:33:36.921958 7566 heartbeater.cc:503] Master 127.2.243.190:36343 was elected leader, sending a full tablet report...
I20260620 11:33:36.922591 7583 raft_consensus.cc:348] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 39ab21cc03214852bd69974973b92cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "becdf81f1c2b4d51a2d39a7c352077e4" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 39051 } } peers { permanent_uuid: "39ab21cc03214852bd69974973b92cac" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 35517 } } peers { permanent_uuid: "7e5c008b2af24a81bf40f01ee8e4c08f" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42095 } }
I20260620 11:33:36.922783 7583 raft_consensus.cc:374] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 39ab21cc03214852bd69974973b92cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:36.922820 7583 raft_consensus.cc:729] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 39ab21cc03214852bd69974973b92cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39ab21cc03214852bd69974973b92cac, State: Initialized, Role: FOLLOWER
I20260620 11:33:36.922971 7583 consensus_queue.cc:260] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 39ab21cc03214852bd69974973b92cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "becdf81f1c2b4d51a2d39a7c352077e4" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 39051 } } peers { permanent_uuid: "39ab21cc03214852bd69974973b92cac" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 35517 } } peers { permanent_uuid: "7e5c008b2af24a81bf40f01ee8e4c08f" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42095 } }
I20260620 11:33:36.923277 7583 ts_tablet_manager.cc:1428] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 39ab21cc03214852bd69974973b92cac: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:33:36.923285 7433 heartbeater.cc:503] Master 127.2.243.190:36343 was elected leader, sending a full tablet report...
I20260620 11:33:36.946499 7586 raft_consensus.cc:482] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:33:36.946617 7586 raft_consensus.cc:504] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "becdf81f1c2b4d51a2d39a7c352077e4" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 39051 } } peers { permanent_uuid: "39ab21cc03214852bd69974973b92cac" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 35517 } } peers { permanent_uuid: "7e5c008b2af24a81bf40f01ee8e4c08f" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42095 } }
I20260620 11:33:36.947070 7586 leader_election.cc:288] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers becdf81f1c2b4d51a2d39a7c352077e4 (127.2.243.131:39051), 39ab21cc03214852bd69974973b92cac (127.2.243.130:35517)
I20260620 11:33:36.951056 7576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:36.947328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:41877 (local address 127.2.243.131:39051)
0620 11:33:36.947375 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:36.947377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:36.947388 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:33:36.947418 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:36.947419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:36.947428 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:36.947491 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:36.947718 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.948475 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.950025 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.950641 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.950708 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:36.950715 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:36.950944 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 11:33:36.950982 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:33:36.951056 7577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:36.947374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:42283 (local address 127.2.243.130:35517)
0620 11:33:36.947430 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:36.947432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:36.947504 (+ 72us) server_negotiation.cc:407] Connection header received
0620 11:33:36.947557 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:36.947558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:36.947574 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:36.947617 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:36.947808 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.948479 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.950024 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.950599 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.950690 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:36.950702 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:36.950943 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 11:33:36.950982 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 11:33:36.951117 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:36.947269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:39051 (local address 127.2.243.129:41877)
0620 11:33:36.947306 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:36.947318 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:36.947360 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:36.947513 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:36.947514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:36.947515 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:36.947636 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:36.947639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.948560 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.948560 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:36.949921 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:36.949924 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.950735 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.950736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:36.950847 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:36.950849 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:33:36.950928 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 11:33:36.950980 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":77}
I20260620 11:33:36.951126 7589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:36.947269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:35517 (local address 127.2.243.129:42283)
0620 11:33:36.947452 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:36.947460 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:36.947507 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:36.947647 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:36.947648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:36.947650 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:36.947731 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:36.947732 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.948559 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.948560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:36.949921 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:36.949924 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.950735 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.950736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:36.950824 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:36.950830 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 11:33:36.950928 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 11:33:36.950980 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":125,"thread_start_us":101,"threads_started":1}
I20260620 11:33:36.951606 7520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0ed0f46bc74ed7a83a9b77f0f99d9c" candidate_uuid: "7e5c008b2af24a81bf40f01ee8e4c08f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "becdf81f1c2b4d51a2d39a7c352077e4" is_pre_election: true
I20260620 11:33:36.951607 7387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0ed0f46bc74ed7a83a9b77f0f99d9c" candidate_uuid: "7e5c008b2af24a81bf40f01ee8e4c08f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39ab21cc03214852bd69974973b92cac" is_pre_election: true
I20260620 11:33:36.951769 7520 raft_consensus.cc:2454] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P becdf81f1c2b4d51a2d39a7c352077e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e5c008b2af24a81bf40f01ee8e4c08f in term 0.
I20260620 11:33:36.951782 7387 raft_consensus.cc:2454] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 39ab21cc03214852bd69974973b92cac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e5c008b2af24a81bf40f01ee8e4c08f in term 0.
I20260620 11:33:36.952024 7187 leader_election.cc:302] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39ab21cc03214852bd69974973b92cac, 7e5c008b2af24a81bf40f01ee8e4c08f; no voters:
I20260620 11:33:36.952164 7586 raft_consensus.cc:2781] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:33:36.952221 7586 raft_consensus.cc:482] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:33:36.952244 7586 raft_consensus.cc:3037] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:36.952934 7586 raft_consensus.cc:504] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "becdf81f1c2b4d51a2d39a7c352077e4" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 39051 } } peers { permanent_uuid: "39ab21cc03214852bd69974973b92cac" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 35517 } } peers { permanent_uuid: "7e5c008b2af24a81bf40f01ee8e4c08f" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42095 } }
I20260620 11:33:36.953076 7586 leader_election.cc:288] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f [CANDIDATE]: Term 1 election: Requested vote from peers becdf81f1c2b4d51a2d39a7c352077e4 (127.2.243.131:39051), 39ab21cc03214852bd69974973b92cac (127.2.243.130:35517)
I20260620 11:33:36.953217 7520 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0ed0f46bc74ed7a83a9b77f0f99d9c" candidate_uuid: "7e5c008b2af24a81bf40f01ee8e4c08f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "becdf81f1c2b4d51a2d39a7c352077e4"
I20260620 11:33:36.953321 7520 raft_consensus.cc:3037] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P becdf81f1c2b4d51a2d39a7c352077e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:36.953353 7387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e0ed0f46bc74ed7a83a9b77f0f99d9c" candidate_uuid: "7e5c008b2af24a81bf40f01ee8e4c08f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39ab21cc03214852bd69974973b92cac"
I20260620 11:33:36.953445 7387 raft_consensus.cc:3037] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 39ab21cc03214852bd69974973b92cac [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:36.954044 7520 raft_consensus.cc:2454] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P becdf81f1c2b4d51a2d39a7c352077e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5c008b2af24a81bf40f01ee8e4c08f in term 1.
I20260620 11:33:36.954187 7387 raft_consensus.cc:2454] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 39ab21cc03214852bd69974973b92cac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e5c008b2af24a81bf40f01ee8e4c08f in term 1.
I20260620 11:33:36.954286 7188 leader_election.cc:302] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e5c008b2af24a81bf40f01ee8e4c08f, becdf81f1c2b4d51a2d39a7c352077e4; no voters:
I20260620 11:33:36.954378 7586 raft_consensus.cc:2781] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:36.954543 7586 raft_consensus.cc:686] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f [term 1 LEADER]: Becoming Leader. State: Replica: 7e5c008b2af24a81bf40f01ee8e4c08f, State: Running, Role: LEADER
I20260620 11:33:36.954653 7586 consensus_queue.cc:237] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "becdf81f1c2b4d51a2d39a7c352077e4" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 39051 } } peers { permanent_uuid: "39ab21cc03214852bd69974973b92cac" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 35517 } } peers { permanent_uuid: "7e5c008b2af24a81bf40f01ee8e4c08f" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42095 } }
I20260620 11:33:36.955482 7109 catalog_manager.cc:5495] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e5c008b2af24a81bf40f01ee8e4c08f (127.2.243.129). New cstate: current_term: 1 leader_uuid: "7e5c008b2af24a81bf40f01ee8e4c08f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "becdf81f1c2b4d51a2d39a7c352077e4" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 39051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "39ab21cc03214852bd69974973b92cac" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 35517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e5c008b2af24a81bf40f01ee8e4c08f" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42095 } health_report { overall_health: HEALTHY } } }
I20260620 11:33:36.969424 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:36.966948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41018 (local address 127.2.243.129:42095)
0620 11:33:36.967004 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:36.967006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:36.967016 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:33:36.967091 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:36.967092 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:36.967132 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:36.967221 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:36.967356 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.967838 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.968606 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:36.968815 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:36.968890 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:36.969068 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:36.969179 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:36.969275 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:36.969300 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 11:33:36.969320 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 11:33:36.970599 7434 tablet.cc:2369] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 39ab21cc03214852bd69974973b92cac: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:33:36.971395 7387 raft_consensus.cc:1261] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 39ab21cc03214852bd69974973b92cac [term 1 FOLLOWER]: Refusing update from remote peer 7e5c008b2af24a81bf40f01ee8e4c08f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:36.971396 7520 raft_consensus.cc:1261] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P becdf81f1c2b4d51a2d39a7c352077e4 [term 1 FOLLOWER]: Refusing update from remote peer 7e5c008b2af24a81bf40f01ee8e4c08f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:36.971779 7590 consensus_queue.cc:1035] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f [LEADER]: Connected to new peer: Peer: permanent_uuid: "39ab21cc03214852bd69974973b92cac" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 35517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:36.971935 7586 consensus_queue.cc:1035] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 7e5c008b2af24a81bf40f01ee8e4c08f [LEADER]: Connected to new peer: Peer: permanent_uuid: "becdf81f1c2b4d51a2d39a7c352077e4" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 39051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:36.974731 7594 mvcc.cc:205] Tried to move back new op lower bound from 7298888568711417856 to 7298888568646643712. Current Snapshot: MvccSnapshot[applied={T|T < 7298888568711417856}]
I20260620 11:33:36.974819 7595 mvcc.cc:205] Tried to move back new op lower bound from 7298888568711417856 to 7298888568646643712. Current Snapshot: MvccSnapshot[applied={T|T < 7298888568711417856}]
I20260620 11:33:36.976645 7109 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:36.976748 7109 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51800:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:36.977496 7109 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c135676571fb426da40c3425a694cfcb: Sending DeleteTablet for 3 replicas of tablet 2e0ed0f46bc74ed7a83a9b77f0f99d9c
I20260620 11:33:36.977829 7234 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e0ed0f46bc74ed7a83a9b77f0f99d9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:36 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:41006
I20260620 11:33:36.977830 7367 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e0ed0f46bc74ed7a83a9b77f0f99d9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:36 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:60214
I20260620 11:33:36.978211 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 7172
I20260620 11:33:36.978216 7500 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e0ed0f46bc74ed7a83a9b77f0f99d9c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:36 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:36068
I20260620 11:33:36.978775 7602 tablet_replica.cc:330] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 39ab21cc03214852bd69974973b92cac: stopping tablet replica
I20260620 11:33:36.978894 7602 raft_consensus.cc:2229] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 39ab21cc03214852bd69974973b92cac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:36.978922 7604 tablet_replica.cc:330] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P becdf81f1c2b4d51a2d39a7c352077e4: stopping tablet replica
I20260620 11:33:36.978998 7602 raft_consensus.cc:2258] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 39ab21cc03214852bd69974973b92cac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:36.979014 7604 raft_consensus.cc:2229] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P becdf81f1c2b4d51a2d39a7c352077e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:36.979110 7604 raft_consensus.cc:2258] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P becdf81f1c2b4d51a2d39a7c352077e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:36.979444 7602 ts_tablet_manager.cc:1905] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 39ab21cc03214852bd69974973b92cac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:36.979550 7604 ts_tablet_manager.cc:1905] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P becdf81f1c2b4d51a2d39a7c352077e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:36.981278 7602 ts_tablet_manager.cc:1918] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 39ab21cc03214852bd69974973b92cac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:33:36.981369 7602 log.cc:1196] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 39ab21cc03214852bd69974973b92cac: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-1/wal/wals/2e0ed0f46bc74ed7a83a9b77f0f99d9c
I20260620 11:33:36.981423 7604 ts_tablet_manager.cc:1918] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P becdf81f1c2b4d51a2d39a7c352077e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:33:36.981500 7604 log.cc:1196] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P becdf81f1c2b4d51a2d39a7c352077e4: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/2e0ed0f46bc74ed7a83a9b77f0f99d9c
I20260620 11:33:36.981653 7602 ts_tablet_manager.cc:1939] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P 39ab21cc03214852bd69974973b92cac: Deleting consensus metadata
I20260620 11:33:36.981770 7604 ts_tablet_manager.cc:1939] T 2e0ed0f46bc74ed7a83a9b77f0f99d9c P becdf81f1c2b4d51a2d39a7c352077e4: Deleting consensus metadata
I20260620 11:33:36.982101 7094 catalog_manager.cc:4841] TS 39ab21cc03214852bd69974973b92cac (127.2.243.130:35517): tablet 2e0ed0f46bc74ed7a83a9b77f0f99d9c (table test-table [id=25ed378080e94eb29392ac897d3dd5d3]) successfully deleted
I20260620 11:33:36.982146 7095 catalog_manager.cc:4841] TS becdf81f1c2b4d51a2d39a7c352077e4 (127.2.243.131:39051): tablet 2e0ed0f46bc74ed7a83a9b77f0f99d9c (table test-table [id=25ed378080e94eb29392ac897d3dd5d3]) successfully deleted
W20260620 11:33:36.984208 7095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.243.129:42095 (error 108)
I20260620 11:33:36.984690 7575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:36.984542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:42095 (local address 127.0.0.1:41032)
0620 11:33:36.984602 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:36.984650 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.129:42095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 11:33:36.984709 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 7305
W20260620 11:33:36.985570 7095 catalog_manager.cc:4584] TS 7e5c008b2af24a81bf40f01ee8e4c08f (127.2.243.129:42095): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2e0ed0f46bc74ed7a83a9b77f0f99d9c: Network error: Client connection negotiation failed: client connection to 127.2.243.129:42095: connect: Connection refused (error 111)
I20260620 11:33:36.989760 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 7438
I20260620 11:33:36.994408 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 7079
2026-06-20T11:33:36Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1927 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T11:33:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:33:37Z Disabled control of system clock
I20260620 11:33:37.021586 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:41053
--webserver_interface=127.2.243.190
--webserver_port=0
--builtin_ntp_servers=127.2.243.148:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:41053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 11:33:37.101653 7609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:37.101869 7609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:37.101891 7609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:37.103154 7609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:33:37.103201 7609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:37.103214 7609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:37.103229 7609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:33:37.103241 7609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:33:37.104640 7609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:41053
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:41053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:37.104915 7609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:37.105237 7609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:37.110407 7617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:37.110399 7614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:37.110704 7609 server_base.cc:1034] running on GCE node
W20260620 11:33:37.110397 7615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:37.111102 7609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:37.111423 7609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:37.112609 7609 hybrid_clock.cc:648] HybridClock initialized: now 1781955217112586 us; error 37 us; skew 500 ppm
I20260620 11:33:37.113992 7609 webserver.cc:460] Webserver started at http://127.2.243.190:45045/ using document root <none> and password file <none>
I20260620 11:33:37.114257 7609 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:37.114342 7609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:37.114493 7609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:37.115665 7609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "542fdfaf63d74067aac9e5208977f53a"
format_stamp: "Formatted at 2026-06-20 11:33:37 on dist-test-slave-ctbw"
server_key: "fbc4da98868e041cfe86dd265fe65173"
server_key_iv: "d7e7e1856a6e8017b1a296d1478cc509"
server_key_version: "encryptionkey@0"
I20260620 11:33:37.116032 7609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "542fdfaf63d74067aac9e5208977f53a"
format_stamp: "Formatted at 2026-06-20 11:33:37 on dist-test-slave-ctbw"
server_key: "fbc4da98868e041cfe86dd265fe65173"
server_key_iv: "d7e7e1856a6e8017b1a296d1478cc509"
server_key_version: "encryptionkey@0"
I20260620 11:33:37.117326 7609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:33:37.118094 7623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:37.118294 7609 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:37.118397 7609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "542fdfaf63d74067aac9e5208977f53a"
format_stamp: "Formatted at 2026-06-20 11:33:37 on dist-test-slave-ctbw"
server_key: "fbc4da98868e041cfe86dd265fe65173"
server_key_iv: "d7e7e1856a6e8017b1a296d1478cc509"
server_key_version: "encryptionkey@0"
I20260620 11:33:37.118470 7609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:37.121845 7609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:37.122108 7609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:37.122253 7609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:37.126260 7609 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:41053
I20260620 11:33:37.126323 7675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:41053 every 8 connection(s)
I20260620 11:33:37.126654 7609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:33:37.127344 7676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:37.130052 7676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a: Bootstrap starting.
I20260620 11:33:37.130628 7676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:37.130965 7676 log.cc:824] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:37.131626 7676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a: No bootstrap required, opened a new log
I20260620 11:33:37.133158 7676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542fdfaf63d74067aac9e5208977f53a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41053 } }
I20260620 11:33:37.133291 7676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:37.133312 7676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 542fdfaf63d74067aac9e5208977f53a, State: Initialized, Role: FOLLOWER
I20260620 11:33:37.133476 7676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542fdfaf63d74067aac9e5208977f53a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41053 } }
I20260620 11:33:37.133535 7676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:33:37.133556 7676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:33:37.133623 7676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:37.134145 7676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542fdfaf63d74067aac9e5208977f53a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41053 } }
I20260620 11:33:37.134227 7676 leader_election.cc:302] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 542fdfaf63d74067aac9e5208977f53a; no voters:
I20260620 11:33:37.134409 7676 leader_election.cc:288] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:33:37.134510 7679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:37.134647 7676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:37.134706 7679 raft_consensus.cc:686] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a [term 1 LEADER]: Becoming Leader. State: Replica: 542fdfaf63d74067aac9e5208977f53a, State: Running, Role: LEADER
I20260620 11:33:37.134802 7679 consensus_queue.cc:237] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542fdfaf63d74067aac9e5208977f53a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41053 } }
I20260620 11:33:37.135424 7680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "542fdfaf63d74067aac9e5208977f53a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542fdfaf63d74067aac9e5208977f53a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41053 } } }
I20260620 11:33:37.135521 7680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:37.135932 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 7609
I20260620 11:33:37.136042 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:33:37.136234 3022 external_mini_cluster.cc:1428] Setting key d1eef0b2aca42e36d4acf70c75cc7b59
I20260620 11:33:37.136027 7681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 542fdfaf63d74067aac9e5208977f53a. Latest consensus state: current_term: 1 leader_uuid: "542fdfaf63d74067aac9e5208977f53a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "542fdfaf63d74067aac9e5208977f53a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41053 } } }
I20260620 11:33:37.136022 7688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:33:37.136293 7681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:37.136902 7688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:33:37.138577 7688 catalog_manager.cc:1347] Generated new cluster ID: b630a483803a4597af42705c74b52d1e
I20260620 11:33:37.138648 7688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:33:37.138777 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:37.136729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55806 (local address 127.2.243.190:41053)
0620 11:33:37.136992 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:37.137001 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:37.137036 (+ 35us) server_negotiation.cc:407] Connection header received
0620 11:33:37.137178 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:37.137185 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:37.137230 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:37.137376 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:37.138212 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:37.138215 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:37.138215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:33:37.138218 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:37.138255 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:33:37.138265 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:37.138337 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:37.138483 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 11:33:37.138539 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":152,"threads_started":1}
I20260620 11:33:37.166580 7688 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:33:37.166805 7688 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:33:37.170459 7688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a: Generated new TSK 0
I20260620 11:33:37.170722 7688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:33:37.176179 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:41053
--builtin_ntp_servers=127.2.243.148:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 11:33:37.262029 7700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:37.262233 7700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:37.262254 7700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:37.263479 7700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:37.263523 7700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:37.263550 7700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:33:37.264914 7700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:41053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7700
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:37.265190 7700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:37.265501 7700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:37.271483 7705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:37.271493 7706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:37.271725 7700 server_base.cc:1034] running on GCE node
W20260620 11:33:37.271493 7708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:37.272027 7700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:37.272297 7700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:37.273466 7700 hybrid_clock.cc:648] HybridClock initialized: now 1781955217273450 us; error 32 us; skew 500 ppm
I20260620 11:33:37.274953 7700 webserver.cc:460] Webserver started at http://127.2.243.129:35123/ using document root <none> and password file <none>
I20260620 11:33:37.275238 7700 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:37.275309 7700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:37.275472 7700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:37.276551 7700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "625cb732ac1e4b869d3325ae5d43690d"
format_stamp: "Formatted at 2026-06-20 11:33:37 on dist-test-slave-ctbw"
server_key: "0efe8728880d0f4a95e7a41db73130df"
server_key_iv: "af8ba849a4724bb13de0eb62f9d043e1"
server_key_version: "encryptionkey@0"
I20260620 11:33:37.276952 7700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "625cb732ac1e4b869d3325ae5d43690d"
format_stamp: "Formatted at 2026-06-20 11:33:37 on dist-test-slave-ctbw"
server_key: "0efe8728880d0f4a95e7a41db73130df"
server_key_iv: "af8ba849a4724bb13de0eb62f9d043e1"
server_key_version: "encryptionkey@0"
I20260620 11:33:37.278328 7700 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:33:37.279110 7714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:37.279337 7700 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:37.279438 7700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "625cb732ac1e4b869d3325ae5d43690d"
format_stamp: "Formatted at 2026-06-20 11:33:37 on dist-test-slave-ctbw"
server_key: "0efe8728880d0f4a95e7a41db73130df"
server_key_iv: "af8ba849a4724bb13de0eb62f9d043e1"
server_key_version: "encryptionkey@0"
I20260620 11:33:37.279508 7700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:37.284257 7700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:37.284479 7700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:37.284574 7700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:37.284724 7700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:37.284976 7700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:37.285004 7700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:37.285025 7700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:37.285045 7700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:37.290495 7700 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:35813
I20260620 11:33:37.290577 7827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:35813 every 8 connection(s)
I20260620 11:33:37.290911 7700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:33:37.291328 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 7700
I20260620 11:33:37.291428 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:33:37.291685 3022 external_mini_cluster.cc:1428] Setting key 24d4ad02a2272560bfcd8e379d1b1af5
I20260620 11:33:37.292833 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:41053
--builtin_ntp_servers=127.2.243.148:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 11:33:37.294565 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:37.291768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:35161 (local address 127.2.243.190:41053)
0620 11:33:37.291830 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:37.291832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:37.292122 (+ 290us) server_negotiation.cc:407] Connection header received
0620 11:33:37.292748 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:37.292749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:37.292754 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:37.292795 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:37.294222 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:37.294224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:37.294224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:33:37.294229 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:37.294251 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:33:37.294256 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:37.294337 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:37.294456 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 11:33:37.294475 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 11:33:37.294797 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:37.291760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41053 (local address 127.2.243.129:35161)
0620 11:33:37.291962 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:37.291997 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:37.292642 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:37.292881 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:37.292886 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:37.292897 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:37.294025 (+ 1128us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:33:37.294026 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:37.294075 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:33:37.294115 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:37.294361 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:37.294369 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:37.294372 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:33:37.294457 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 11:33:37.294522 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":83,"threads_started":1}
I20260620 11:33:37.295439 7828 heartbeater.cc:347] Connected to a master server at 127.2.243.190:41053
I20260620 11:33:37.295539 7828 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:37.295701 7828 heartbeater.cc:511] Master 127.2.243.190:41053 requested a full tablet report, sending...
I20260620 11:33:37.296298 7640 ts_manager.cc:194] Registered new tserver with Master: 625cb732ac1e4b869d3325ae5d43690d (127.2.243.129:35813)
W20260620 11:33:37.376801 7831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:37.377043 7831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:37.377066 7831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:37.378188 7831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:37.378221 7831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:37.378247 7831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:33:37.379671 7831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:41053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:37.379870 7831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:37.380120 7831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:37.385394 7837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:37.385394 7839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:37.385409 7836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:37.385898 7831 server_base.cc:1034] running on GCE node
I20260620 11:33:37.386066 7831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:37.386272 7831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:37.387418 7831 hybrid_clock.cc:648] HybridClock initialized: now 1781955217387401 us; error 29 us; skew 500 ppm
I20260620 11:33:37.388692 7831 webserver.cc:460] Webserver started at http://127.2.243.130:41795/ using document root <none> and password file <none>
I20260620 11:33:37.388957 7831 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:37.389009 7831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:37.389189 7831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:37.390246 7831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "c5df47a991884ae198785de12b7915bb"
format_stamp: "Formatted at 2026-06-20 11:33:37 on dist-test-slave-ctbw"
server_key: "17ce22bf0caac2cba79e9541ae12ce5a"
server_key_iv: "ef065719589adc7d50f8393a5251e791"
server_key_version: "encryptionkey@0"
I20260620 11:33:37.390597 7831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "c5df47a991884ae198785de12b7915bb"
format_stamp: "Formatted at 2026-06-20 11:33:37 on dist-test-slave-ctbw"
server_key: "17ce22bf0caac2cba79e9541ae12ce5a"
server_key_iv: "ef065719589adc7d50f8393a5251e791"
server_key_version: "encryptionkey@0"
I20260620 11:33:37.392025 7831 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:33:37.392835 7845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:37.393018 7831 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:37.393119 7831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "c5df47a991884ae198785de12b7915bb"
format_stamp: "Formatted at 2026-06-20 11:33:37 on dist-test-slave-ctbw"
server_key: "17ce22bf0caac2cba79e9541ae12ce5a"
server_key_iv: "ef065719589adc7d50f8393a5251e791"
server_key_version: "encryptionkey@0"
I20260620 11:33:37.393201 7831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:37.396620 7831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:37.396875 7831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:37.397027 7831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:37.397229 7831 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:37.397531 7831 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:37.397583 7831 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:37.397630 7831 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:37.397657 7831 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:37.403602 7831 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:44023
I20260620 11:33:37.403643 7958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:44023 every 8 connection(s)
I20260620 11:33:37.403971 7831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:33:37.406798 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:37.404704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:41965 (local address 127.2.243.190:41053)
0620 11:33:37.404751 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:37.404753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:37.404945 (+ 192us) server_negotiation.cc:407] Connection header received
0620 11:33:37.405395 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:37.405396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:37.405401 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:37.405467 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:37.406557 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:37.406558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:37.406558 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:33:37.406563 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:37.406577 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:33:37.406581 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:37.406640 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:37.406735 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 11:33:37.406748 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:33:37.406988 7961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:37.404635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41053 (local address 127.2.243.130:41965)
0620 11:33:37.404808 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:37.404859 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:37.405314 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:37.405524 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:37.405527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:37.405535 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:37.406385 (+ 850us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:33:37.406385 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:37.406411 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:33:37.406439 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:37.406662 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:37.406665 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:37.406668 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:33:37.406727 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 11:33:37.406773 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
I20260620 11:33:37.407528 7959 heartbeater.cc:347] Connected to a master server at 127.2.243.190:41053
I20260620 11:33:37.407675 7959 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:37.407845 7959 heartbeater.cc:511] Master 127.2.243.190:41053 requested a full tablet report, sending...
I20260620 11:33:37.408246 7640 ts_manager.cc:194] Registered new tserver with Master: c5df47a991884ae198785de12b7915bb (127.2.243.130:44023)
I20260620 11:33:37.409027 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 7831
I20260620 11:33:37.409118 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:33:37.409268 3022 external_mini_cluster.cc:1428] Setting key 3de408952680e8e18db4bf6b8438e470
I20260620 11:33:37.410080 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:41053
--builtin_ntp_servers=127.2.243.148:36091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 11:33:37.490991 7962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:37.491194 7962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:37.491218 7962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:37.492352 7962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:37.492386 7962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:37.492412 7962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:33:37.493700 7962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:36091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:41053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:37.493889 7962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:37.494177 7962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:37.499270 7968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:37.499265 7967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:37.499269 7970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:37.499554 7962 server_base.cc:1034] running on GCE node
I20260620 11:33:37.499740 7962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:37.499925 7962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:37.501068 7962 hybrid_clock.cc:648] HybridClock initialized: now 1781955217501056 us; error 26 us; skew 500 ppm
I20260620 11:33:37.502339 7962 webserver.cc:460] Webserver started at http://127.2.243.131:38993/ using document root <none> and password file <none>
I20260620 11:33:37.502539 7962 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:37.502580 7962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:37.502714 7962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:37.503793 7962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "4bdb0fb49b3d4af99961a88dbbf6bf38"
format_stamp: "Formatted at 2026-06-20 11:33:37 on dist-test-slave-ctbw"
server_key: "a426eaf0cd54f2dbf2e04a585ce0d059"
server_key_iv: "0a81f79eec3930206fb9e6cda5120218"
server_key_version: "encryptionkey@0"
I20260620 11:33:37.504096 7962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "4bdb0fb49b3d4af99961a88dbbf6bf38"
format_stamp: "Formatted at 2026-06-20 11:33:37 on dist-test-slave-ctbw"
server_key: "a426eaf0cd54f2dbf2e04a585ce0d059"
server_key_iv: "0a81f79eec3930206fb9e6cda5120218"
server_key_version: "encryptionkey@0"
I20260620 11:33:37.505401 7962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:33:37.506117 7976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:37.506300 7962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:37.506368 7962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "4bdb0fb49b3d4af99961a88dbbf6bf38"
format_stamp: "Formatted at 2026-06-20 11:33:37 on dist-test-slave-ctbw"
server_key: "a426eaf0cd54f2dbf2e04a585ce0d059"
server_key_iv: "0a81f79eec3930206fb9e6cda5120218"
server_key_version: "encryptionkey@0"
I20260620 11:33:37.506420 7962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:37.509795 7962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:37.510015 7962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:37.510102 7962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:37.510248 7962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:37.510474 7962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:37.510504 7962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:37.510522 7962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:37.510535 7962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:37.516330 7962 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:38557
I20260620 11:33:37.516371 8089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:38557 every 8 connection(s)
I20260620 11:33:37.516683 7962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:33:37.519750 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:37.517345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:44815 (local address 127.2.243.190:41053)
0620 11:33:37.517395 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:37.517396 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:37.517641 (+ 245us) server_negotiation.cc:407] Connection header received
0620 11:33:37.518174 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:37.518176 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:37.518181 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:37.518244 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:37.519482 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:37.519484 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:37.519484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:33:37.519488 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:37.519503 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:33:37.519508 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:37.519569 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:37.519665 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 11:33:37.519680 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 11:33:37.519944 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:37.517339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41053 (local address 127.2.243.131:44815)
0620 11:33:37.517526 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:37.517557 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:37.518060 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:37.518327 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:37.518330 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:37.518338 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:37.519330 (+ 992us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:33:37.519331 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:37.519358 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:33:37.519385 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:37.519570 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:37.519575 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:37.519578 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:33:37.519678 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 11:33:37.519738 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260620 11:33:37.520466 8090 heartbeater.cc:347] Connected to a master server at 127.2.243.190:41053
I20260620 11:33:37.520561 8090 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:37.520732 8090 heartbeater.cc:511] Master 127.2.243.190:41053 requested a full tablet report, sending...
I20260620 11:33:37.521106 7640 ts_manager.cc:194] Registered new tserver with Master: 4bdb0fb49b3d4af99961a88dbbf6bf38 (127.2.243.131:38557)
I20260620 11:33:37.524682 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 7962
I20260620 11:33:37.524768 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:33:37.524914 3022 external_mini_cluster.cc:1428] Setting key 8e0cc0dae77ed8f1d8ca607276cafa73
I20260620 11:33:37.525806 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:33:37.528139 7698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:37.527127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55822 (local address 127.2.243.190:41053)
0620 11:33:37.527211 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:37.527213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:37.527309 (+ 96us) server_negotiation.cc:407] Connection header received
0620 11:33:37.527357 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:37.527358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:37.527362 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:37.527410 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:37.527927 (+ 517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:37.527928 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:37.527928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:33:37.527931 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:37.527946 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:33:37.527950 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:37.527993 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:37.528029 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 11:33:37.528041 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 11:33:37.529516 7640 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:33:37.531052 7640 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:33:37.535173 8100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:37.533302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:38557 (local address 127.0.0.1:41420)
0620 11:33:37.533503 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:37.533546 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:37.533606 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:37.534166 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:37.534166 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:37.534169 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:37.534602 (+ 433us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:33:37.534603 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:37.534615 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:33:37.534670 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:37.534996 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:37.535002 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:37.535005 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:33:37.535084 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 11:33:37.535098 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":78,"threads_started":1}
I20260620 11:33:37.535359 8103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:37.533679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41420 (local address 127.2.243.131:38557)
0620 11:33:37.533840 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:37.533844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:37.533853 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:33:37.533878 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:37.533880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:37.533887 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:37.533953 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:37.534755 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:37.534761 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:37.534761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:33:37.534765 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:37.534820 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:33:37.534846 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:37.535004 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:37.535108 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 11:33:37.535126 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260620 11:33:37.535396 8098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:37.533302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:35813 (local address 127.0.0.1:50408)
0620 11:33:37.533710 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:37.533721 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:37.533772 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:37.533944 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:37.533950 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:37.533958 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:37.534851 (+ 893us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:33:37.534852 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:37.534860 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:33:37.534894 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:37.535196 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:37.535197 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:37.535198 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 11:33:37.535300 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 11:33:37.535314 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":4,"thread_start_us":103,"threads_started":1}
I20260620 11:33:37.535727 8101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:37.533484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50408 (local address 127.2.243.129:35813)
0620 11:33:37.533655 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:37.533660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:37.533789 (+ 129us) server_negotiation.cc:407] Connection header received
0620 11:33:37.533821 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:37.533824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:37.533833 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:37.533927 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:37.534986 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:37.534995 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:37.534995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:33:37.535000 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:37.535052 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:33:37.535094 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:37.535158 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:37.535312 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 11:33:37.535331 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260620 11:33:37.535725 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:37.533332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:44023 (local address 127.0.0.1:38960)
0620 11:33:37.533865 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:37.533873 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:37.533970 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:37.534536 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:37.534537 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:37.534540 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:37.535046 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:33:37.535046 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:37.535052 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:33:37.535063 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:37.535564 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:37.535565 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:37.535565 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 11:33:37.535651 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 11:33:37.535667 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":74,"threads_started":1}
I20260620 11:33:37.536114 8024 tablet_service.cc:1467] Processing CreateTablet for tablet e09e00ad82784983a5a711f6db88a7f2 (DEFAULT_TABLE table=test-table [id=ceb37b710c21413b9ffeca05754c2cde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:37.536356 7762 tablet_service.cc:1467] Processing CreateTablet for tablet e09e00ad82784983a5a711f6db88a7f2 (DEFAULT_TABLE table=test-table [id=ceb37b710c21413b9ffeca05754c2cde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:37.536545 8024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e09e00ad82784983a5a711f6db88a7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:37.536718 7762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e09e00ad82784983a5a711f6db88a7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:37.539055 8102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:37.533539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38960 (local address 127.2.243.130:44023)
0620 11:33:37.533715 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:37.533718 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:37.533947 (+ 229us) server_negotiation.cc:407] Connection header received
0620 11:33:37.534064 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:37.534069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:37.534080 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:37.534158 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:37.535164 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:37.535191 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:37.535191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:33:37.535196 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:37.535231 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:33:37.535262 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:37.535302 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:37.538960 (+ 3658us) server_negotiation.cc:299] Negotiation successful
0620 11:33:37.538980 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":77,"threads_started":1}
I20260620 11:33:37.539611 7893 tablet_service.cc:1467] Processing CreateTablet for tablet e09e00ad82784983a5a711f6db88a7f2 (DEFAULT_TABLE table=test-table [id=ceb37b710c21413b9ffeca05754c2cde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:37.539942 7893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e09e00ad82784983a5a711f6db88a7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:37.542537 8106 tablet_bootstrap.cc:492] T e09e00ad82784983a5a711f6db88a7f2 P c5df47a991884ae198785de12b7915bb: Bootstrap starting.
I20260620 11:33:37.542631 8104 tablet_bootstrap.cc:492] T e09e00ad82784983a5a711f6db88a7f2 P 625cb732ac1e4b869d3325ae5d43690d: Bootstrap starting.
I20260620 11:33:37.543033 8105 tablet_bootstrap.cc:492] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38: Bootstrap starting.
I20260620 11:33:37.543329 8106 tablet_bootstrap.cc:654] T e09e00ad82784983a5a711f6db88a7f2 P c5df47a991884ae198785de12b7915bb: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:37.543354 8104 tablet_bootstrap.cc:654] T e09e00ad82784983a5a711f6db88a7f2 P 625cb732ac1e4b869d3325ae5d43690d: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:37.543644 8106 log.cc:824] T e09e00ad82784983a5a711f6db88a7f2 P c5df47a991884ae198785de12b7915bb: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:37.543645 8105 tablet_bootstrap.cc:654] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:37.543668 8104 log.cc:824] T e09e00ad82784983a5a711f6db88a7f2 P 625cb732ac1e4b869d3325ae5d43690d: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:37.543955 8105 log.cc:824] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:37.544530 8104 tablet_bootstrap.cc:492] T e09e00ad82784983a5a711f6db88a7f2 P 625cb732ac1e4b869d3325ae5d43690d: No bootstrap required, opened a new log
I20260620 11:33:37.544617 8104 ts_tablet_manager.cc:1397] T e09e00ad82784983a5a711f6db88a7f2 P 625cb732ac1e4b869d3325ae5d43690d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:33:37.544646 8105 tablet_bootstrap.cc:492] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38: No bootstrap required, opened a new log
I20260620 11:33:37.544667 8106 tablet_bootstrap.cc:492] T e09e00ad82784983a5a711f6db88a7f2 P c5df47a991884ae198785de12b7915bb: No bootstrap required, opened a new log
I20260620 11:33:37.544716 8105 ts_tablet_manager.cc:1397] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:33:37.544744 8106 ts_tablet_manager.cc:1397] T e09e00ad82784983a5a711f6db88a7f2 P c5df47a991884ae198785de12b7915bb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 11:33:37.546779 8104 raft_consensus.cc:348] T e09e00ad82784983a5a711f6db88a7f2 P 625cb732ac1e4b869d3325ae5d43690d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cb732ac1e4b869d3325ae5d43690d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 35813 } } peers { permanent_uuid: "4bdb0fb49b3d4af99961a88dbbf6bf38" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38557 } } peers { permanent_uuid: "c5df47a991884ae198785de12b7915bb" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 44023 } }
I20260620 11:33:37.546989 8104 raft_consensus.cc:374] T e09e00ad82784983a5a711f6db88a7f2 P 625cb732ac1e4b869d3325ae5d43690d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:37.547037 8104 raft_consensus.cc:729] T e09e00ad82784983a5a711f6db88a7f2 P 625cb732ac1e4b869d3325ae5d43690d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625cb732ac1e4b869d3325ae5d43690d, State: Initialized, Role: FOLLOWER
I20260620 11:33:37.547158 8104 consensus_queue.cc:260] T e09e00ad82784983a5a711f6db88a7f2 P 625cb732ac1e4b869d3325ae5d43690d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cb732ac1e4b869d3325ae5d43690d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 35813 } } peers { permanent_uuid: "4bdb0fb49b3d4af99961a88dbbf6bf38" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38557 } } peers { permanent_uuid: "c5df47a991884ae198785de12b7915bb" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 44023 } }
I20260620 11:33:37.547457 8104 ts_tablet_manager.cc:1428] T e09e00ad82784983a5a711f6db88a7f2 P 625cb732ac1e4b869d3325ae5d43690d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:33:37.547370 8105 raft_consensus.cc:348] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cb732ac1e4b869d3325ae5d43690d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 35813 } } peers { permanent_uuid: "4bdb0fb49b3d4af99961a88dbbf6bf38" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38557 } } peers { permanent_uuid: "c5df47a991884ae198785de12b7915bb" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 44023 } }
I20260620 11:33:37.547370 8106 raft_consensus.cc:348] T e09e00ad82784983a5a711f6db88a7f2 P c5df47a991884ae198785de12b7915bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cb732ac1e4b869d3325ae5d43690d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 35813 } } peers { permanent_uuid: "4bdb0fb49b3d4af99961a88dbbf6bf38" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38557 } } peers { permanent_uuid: "c5df47a991884ae198785de12b7915bb" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 44023 } }
I20260620 11:33:37.547525 8105 raft_consensus.cc:374] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:37.547525 8106 raft_consensus.cc:374] T e09e00ad82784983a5a711f6db88a7f2 P c5df47a991884ae198785de12b7915bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:37.547569 8105 raft_consensus.cc:729] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bdb0fb49b3d4af99961a88dbbf6bf38, State: Initialized, Role: FOLLOWER
I20260620 11:33:37.547569 8106 raft_consensus.cc:729] T e09e00ad82784983a5a711f6db88a7f2 P c5df47a991884ae198785de12b7915bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5df47a991884ae198785de12b7915bb, State: Initialized, Role: FOLLOWER
I20260620 11:33:37.547597 7828 heartbeater.cc:503] Master 127.2.243.190:41053 was elected leader, sending a full tablet report...
I20260620 11:33:37.547695 8106 consensus_queue.cc:260] T e09e00ad82784983a5a711f6db88a7f2 P c5df47a991884ae198785de12b7915bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cb732ac1e4b869d3325ae5d43690d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 35813 } } peers { permanent_uuid: "4bdb0fb49b3d4af99961a88dbbf6bf38" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38557 } } peers { permanent_uuid: "c5df47a991884ae198785de12b7915bb" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 44023 } }
I20260620 11:33:37.547688 8105 consensus_queue.cc:260] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cb732ac1e4b869d3325ae5d43690d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 35813 } } peers { permanent_uuid: "4bdb0fb49b3d4af99961a88dbbf6bf38" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38557 } } peers { permanent_uuid: "c5df47a991884ae198785de12b7915bb" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 44023 } }
I20260620 11:33:37.547943 8106 ts_tablet_manager.cc:1428] T e09e00ad82784983a5a711f6db88a7f2 P c5df47a991884ae198785de12b7915bb: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 11:33:37.547943 8105 ts_tablet_manager.cc:1428] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 11:33:37.548041 8090 heartbeater.cc:503] Master 127.2.243.190:41053 was elected leader, sending a full tablet report...
I20260620 11:33:37.548069 7959 heartbeater.cc:503] Master 127.2.243.190:41053 was elected leader, sending a full tablet report...
W20260620 11:33:37.654512 7960 tablet.cc:2369] T e09e00ad82784983a5a711f6db88a7f2 P c5df47a991884ae198785de12b7915bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:33:37.751633 8112 raft_consensus.cc:482] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:33:37.751792 8112 raft_consensus.cc:504] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cb732ac1e4b869d3325ae5d43690d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 35813 } } peers { permanent_uuid: "4bdb0fb49b3d4af99961a88dbbf6bf38" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38557 } } peers { permanent_uuid: "c5df47a991884ae198785de12b7915bb" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 44023 } }
I20260620 11:33:37.752197 8112 leader_election.cc:288] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 625cb732ac1e4b869d3325ae5d43690d (127.2.243.129:35813), c5df47a991884ae198785de12b7915bb (127.2.243.130:44023)
I20260620 11:33:37.753599 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:37.752423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:35813 (local address 127.2.243.131:39133)
0620 11:33:37.752466 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:37.752479 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:37.752518 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:37.752753 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:37.752754 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:37.752756 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:37.753221 (+ 465us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:33:37.753222 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:37.753228 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:33:37.753240 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:37.753462 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:37.753463 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:37.753464 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 11:33:37.753518 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 11:33:37.753533 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260620 11:33:37.753599 8101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:37.752541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:39133 (local address 127.2.243.129:35813)
0620 11:33:37.752577 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:37.752579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:37.752587 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:33:37.752615 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:37.752616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:37.752623 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:37.752722 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:37.753350 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:37.753352 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:37.753352 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:33:37.753356 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:37.753373 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:33:37.753377 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:37.753465 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:37.753518 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 11:33:37.753533 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260620 11:33:37.753657 8102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:37.752490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:45763 (local address 127.2.243.130:44023)
0620 11:33:37.752537 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:37.752540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:37.752768 (+ 228us) server_negotiation.cc:407] Connection header received
0620 11:33:37.752789 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:37.752790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:37.752799 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:37.752840 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:37.753408 (+ 568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:37.753410 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:37.753410 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:33:37.753414 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:37.753428 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:33:37.753449 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:37.753515 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:37.753569 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 11:33:37.753583 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:33:37.753747 8113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:37.752423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:44023 (local address 127.2.243.131:45763)
0620 11:33:37.752621 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:37.752635 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:37.752726 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:37.752860 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:37.752861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:37.752863 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:37.753333 (+ 470us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:33:37.753334 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:37.753339 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:33:37.753347 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:37.753511 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:37.753512 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:37.753513 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 11:33:37.753569 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 11:33:37.753583 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":48,"thread_start_us":93,"threads_started":1}
I20260620 11:33:37.754065 7913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09e00ad82784983a5a711f6db88a7f2" candidate_uuid: "4bdb0fb49b3d4af99961a88dbbf6bf38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5df47a991884ae198785de12b7915bb" is_pre_election: true
I20260620 11:33:37.754130 7782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09e00ad82784983a5a711f6db88a7f2" candidate_uuid: "4bdb0fb49b3d4af99961a88dbbf6bf38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625cb732ac1e4b869d3325ae5d43690d" is_pre_election: true
I20260620 11:33:37.754235 7913 raft_consensus.cc:2454] T e09e00ad82784983a5a711f6db88a7f2 P c5df47a991884ae198785de12b7915bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bdb0fb49b3d4af99961a88dbbf6bf38 in term 0.
I20260620 11:33:37.754292 7782 raft_consensus.cc:2454] T e09e00ad82784983a5a711f6db88a7f2 P 625cb732ac1e4b869d3325ae5d43690d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bdb0fb49b3d4af99961a88dbbf6bf38 in term 0.
I20260620 11:33:37.754468 7980 leader_election.cc:302] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bdb0fb49b3d4af99961a88dbbf6bf38, c5df47a991884ae198785de12b7915bb; no voters:
I20260620 11:33:37.754601 8112 raft_consensus.cc:2781] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:33:37.754653 8112 raft_consensus.cc:482] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:33:37.754696 8112 raft_consensus.cc:3037] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:37.755453 8112 raft_consensus.cc:504] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cb732ac1e4b869d3325ae5d43690d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 35813 } } peers { permanent_uuid: "4bdb0fb49b3d4af99961a88dbbf6bf38" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38557 } } peers { permanent_uuid: "c5df47a991884ae198785de12b7915bb" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 44023 } }
I20260620 11:33:37.755599 8112 leader_election.cc:288] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38 [CANDIDATE]: Term 1 election: Requested vote from peers 625cb732ac1e4b869d3325ae5d43690d (127.2.243.129:35813), c5df47a991884ae198785de12b7915bb (127.2.243.130:44023)
I20260620 11:33:37.755777 7913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09e00ad82784983a5a711f6db88a7f2" candidate_uuid: "4bdb0fb49b3d4af99961a88dbbf6bf38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5df47a991884ae198785de12b7915bb"
I20260620 11:33:37.755863 7782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09e00ad82784983a5a711f6db88a7f2" candidate_uuid: "4bdb0fb49b3d4af99961a88dbbf6bf38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "625cb732ac1e4b869d3325ae5d43690d"
I20260620 11:33:37.755882 7913 raft_consensus.cc:3037] T e09e00ad82784983a5a711f6db88a7f2 P c5df47a991884ae198785de12b7915bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:37.755939 7782 raft_consensus.cc:3037] T e09e00ad82784983a5a711f6db88a7f2 P 625cb732ac1e4b869d3325ae5d43690d [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:37.756776 7782 raft_consensus.cc:2454] T e09e00ad82784983a5a711f6db88a7f2 P 625cb732ac1e4b869d3325ae5d43690d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bdb0fb49b3d4af99961a88dbbf6bf38 in term 1.
I20260620 11:33:37.756776 7913 raft_consensus.cc:2454] T e09e00ad82784983a5a711f6db88a7f2 P c5df47a991884ae198785de12b7915bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bdb0fb49b3d4af99961a88dbbf6bf38 in term 1.
I20260620 11:33:37.756944 7980 leader_election.cc:302] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bdb0fb49b3d4af99961a88dbbf6bf38, c5df47a991884ae198785de12b7915bb; no voters:
I20260620 11:33:37.757050 8112 raft_consensus.cc:2781] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:37.757179 8112 raft_consensus.cc:686] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38 [term 1 LEADER]: Becoming Leader. State: Replica: 4bdb0fb49b3d4af99961a88dbbf6bf38, State: Running, Role: LEADER
I20260620 11:33:37.757273 8112 consensus_queue.cc:237] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cb732ac1e4b869d3325ae5d43690d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 35813 } } peers { permanent_uuid: "4bdb0fb49b3d4af99961a88dbbf6bf38" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38557 } } peers { permanent_uuid: "c5df47a991884ae198785de12b7915bb" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 44023 } }
I20260620 11:33:37.758041 7640 catalog_manager.cc:5495] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bdb0fb49b3d4af99961a88dbbf6bf38 (127.2.243.131). New cstate: current_term: 1 leader_uuid: "4bdb0fb49b3d4af99961a88dbbf6bf38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "625cb732ac1e4b869d3325ae5d43690d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 35813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bdb0fb49b3d4af99961a88dbbf6bf38" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 38557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5df47a991884ae198785de12b7915bb" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 44023 } health_report { overall_health: UNKNOWN } } }
I20260620 11:33:37.761620 8103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:37.760582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.2.243.131:38557)
0620 11:33:37.760621 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:37.760622 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:37.760754 (+ 132us) server_negotiation.cc:407] Connection header received
0620 11:33:37.760809 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:37.760809 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:37.760819 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:37.760872 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:37.761453 (+ 581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:37.761454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:37.761455 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:33:37.761458 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:37.761472 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:33:37.761476 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:37.761534 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:37.761566 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 11:33:37.761577 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:33:37.763294 7913 raft_consensus.cc:1261] T e09e00ad82784983a5a711f6db88a7f2 P c5df47a991884ae198785de12b7915bb [term 1 FOLLOWER]: Refusing update from remote peer 4bdb0fb49b3d4af99961a88dbbf6bf38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:37.763294 7782 raft_consensus.cc:1261] T e09e00ad82784983a5a711f6db88a7f2 P 625cb732ac1e4b869d3325ae5d43690d [term 1 FOLLOWER]: Refusing update from remote peer 4bdb0fb49b3d4af99961a88dbbf6bf38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:37.763706 8112 consensus_queue.cc:1035] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5df47a991884ae198785de12b7915bb" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 44023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:37.763855 8114 consensus_queue.cc:1035] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "625cb732ac1e4b869d3325ae5d43690d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 35813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:37.766109 8117 mvcc.cc:205] Tried to move back new op lower bound from 7298888571955728384 to 7298888571934326784. Current Snapshot: MvccSnapshot[applied={T|T < 7298888571955728384}]
I20260620 11:33:37.766623 8118 mvcc.cc:205] Tried to move back new op lower bound from 7298888571955728384 to 7298888571934326784. Current Snapshot: MvccSnapshot[applied={T|T < 7298888571955728384}]
I20260620 11:33:37.767464 8119 mvcc.cc:205] Tried to move back new op lower bound from 7298888571955728384 to 7298888571934326784. Current Snapshot: MvccSnapshot[applied={T|T < 7298888571955728384 or (T in {7298888571955728384})}]
I20260620 11:33:37.768615 7638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:37.768756 7638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:37.769531 7638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 542fdfaf63d74067aac9e5208977f53a: Sending DeleteTablet for 3 replicas of tablet e09e00ad82784983a5a711f6db88a7f2
I20260620 11:33:37.769846 7762 tablet_service.cc:1514] Processing DeleteTablet for tablet e09e00ad82784983a5a711f6db88a7f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:37 UTC) from {username='slave'} at 127.0.0.1:50408
I20260620 11:33:37.769889 7893 tablet_service.cc:1514] Processing DeleteTablet for tablet e09e00ad82784983a5a711f6db88a7f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:37 UTC) from {username='slave'} at 127.0.0.1:38960
I20260620 11:33:37.769914 8024 tablet_service.cc:1514] Processing DeleteTablet for tablet e09e00ad82784983a5a711f6db88a7f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:37 UTC) from {username='slave'} at 127.0.0.1:41420
I20260620 11:33:37.770160 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 7700
I20260620 11:33:37.770985 8129 tablet_replica.cc:330] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38: stopping tablet replica
I20260620 11:33:37.771135 8129 raft_consensus.cc:2229] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:33:37.771334 8129 raft_consensus.cc:2258] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:37.771785 8129 ts_tablet_manager.cc:1905] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:37.773442 8129 ts_tablet_manager.cc:1918] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:33:37.773566 8129 log.cc:1196] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/e09e00ad82784983a5a711f6db88a7f2
I20260620 11:33:37.773900 8129 ts_tablet_manager.cc:1939] T e09e00ad82784983a5a711f6db88a7f2 P 4bdb0fb49b3d4af99961a88dbbf6bf38: Deleting consensus metadata
I20260620 11:33:37.774348 7626 catalog_manager.cc:4841] TS 4bdb0fb49b3d4af99961a88dbbf6bf38 (127.2.243.131:38557): tablet e09e00ad82784983a5a711f6db88a7f2 (table test-table [id=ceb37b710c21413b9ffeca05754c2cde]) successfully deleted
W20260620 11:33:37.774961 7624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.243.129:35813 (error 108)
W20260620 11:33:37.776019 8099 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:37.775307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:35813 (local address 127.0.0.1:50414)
0620 11:33:37.775365 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:37.775378 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:37.775418 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:37.775964 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.129:35813: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 11:33:37.776211 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 7831
I20260620 11:33:37.776240 8128 tablet_replica.cc:330] T e09e00ad82784983a5a711f6db88a7f2 P c5df47a991884ae198785de12b7915bb: stopping tablet replica
W20260620 11:33:37.776253 7624 catalog_manager.cc:4584] TS 625cb732ac1e4b869d3325ae5d43690d (127.2.243.129:35813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e09e00ad82784983a5a711f6db88a7f2: Network error: Client connection negotiation failed: client connection to 127.2.243.129:35813: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 11:33:37.781031 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:37.780901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:44023 (local address 127.0.0.1:38966)
0620 11:33:37.780956 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:37.780996 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.130:44023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 11:33:37.781078 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 7962
I20260620 11:33:37.785625 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 7609
2026-06-20T11:33:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (790 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 11:33:37.805099 3022 mini_oidc.cc:152] Starting JWKS server
I20260620 11:33:37.806185 3022 webserver.cc:460] Webserver started at https://127.0.0.1:35211/ using document root <none> and password file <none>
I20260620 11:33:37.806306 3022 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 11:33:37.806495 3022 webserver.cc:460] Webserver started at http://127.0.0.1:37555/ using document root <none> and password file <none>
2026-06-20T11:33:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:33:37Z Disabled control of system clock
I20260620 11:33:37.814064 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:33677
--webserver_interface=127.2.243.190
--webserver_port=0
--builtin_ntp_servers=127.2.243.148:41439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:33677
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35211/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955192827595-3022-0/testchainca.cert with env {}
W20260620 11:33:37.896045 8136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:37.896246 8136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:37.896270 8136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:37.897969 8136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:33:37.898010 8136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:37.898023 8136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:37.898036 8136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:33:37.898077 8136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 11:33:37.898099 8136 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 11:33:37.898124 8136 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35211/jwks.json
I20260620 11:33:37.900197 8136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:41439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:33677
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:33677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35211/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955192827595-3022-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:37.900463 8136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:37.900719 8136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:37.905591 8142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:37.905615 8144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:37.905601 8141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:37.905617 8136 server_base.cc:1034] running on GCE node
I20260620 11:33:37.906105 8136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:37.906385 8136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:37.907552 8136 hybrid_clock.cc:648] HybridClock initialized: now 1781955217907535 us; error 34 us; skew 500 ppm
I20260620 11:33:37.908854 8136 webserver.cc:460] Webserver started at http://127.2.243.190:34679/ using document root <none> and password file <none>
I20260620 11:33:37.909091 8136 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:37.909155 8136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:37.909323 8136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:37.910343 8136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "ce3f521ef3d843beb50dd1217da821d9"
format_stamp: "Formatted at 2026-06-20 11:33:37 on dist-test-slave-ctbw"
server_key: "0862560350d2014a1ea73a5df6f16477"
server_key_iv: "8b90be678dc7c2a3242bbb2fe81bbf03"
server_key_version: "encryptionkey@0"
I20260620 11:33:37.910635 8136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "ce3f521ef3d843beb50dd1217da821d9"
format_stamp: "Formatted at 2026-06-20 11:33:37 on dist-test-slave-ctbw"
server_key: "0862560350d2014a1ea73a5df6f16477"
server_key_iv: "8b90be678dc7c2a3242bbb2fe81bbf03"
server_key_version: "encryptionkey@0"
I20260620 11:33:37.911933 8136 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:33:37.912636 8150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:37.912806 8136 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:37.912876 8136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "ce3f521ef3d843beb50dd1217da821d9"
format_stamp: "Formatted at 2026-06-20 11:33:37 on dist-test-slave-ctbw"
server_key: "0862560350d2014a1ea73a5df6f16477"
server_key_iv: "8b90be678dc7c2a3242bbb2fe81bbf03"
server_key_version: "encryptionkey@0"
I20260620 11:33:37.912930 8136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:37.945834 8136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:37.946130 8136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:37.946233 8136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:37.950280 8136 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:33677
I20260620 11:33:37.950320 8202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:33677 every 8 connection(s)
I20260620 11:33:37.950698 8136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:33:37.951661 8203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:37.953961 8203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9: Bootstrap starting.
I20260620 11:33:37.954576 8203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:37.954936 8203 log.cc:824] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:37.955629 8203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9: No bootstrap required, opened a new log
I20260620 11:33:37.957166 8203 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce3f521ef3d843beb50dd1217da821d9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33677 } }
I20260620 11:33:37.957322 8203 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:37.957345 8203 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce3f521ef3d843beb50dd1217da821d9, State: Initialized, Role: FOLLOWER
I20260620 11:33:37.957512 8203 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce3f521ef3d843beb50dd1217da821d9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33677 } }
I20260620 11:33:37.957583 8203 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:33:37.957605 8203 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:33:37.957652 8203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:37.958151 8203 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce3f521ef3d843beb50dd1217da821d9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33677 } }
I20260620 11:33:37.958244 8203 leader_election.cc:302] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce3f521ef3d843beb50dd1217da821d9; no voters:
I20260620 11:33:37.958396 8203 leader_election.cc:288] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:33:37.958477 8206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:37.958506 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 8136
I20260620 11:33:37.958607 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:33:37.958662 8203 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:37.958693 8206 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9 [term 1 LEADER]: Becoming Leader. State: Replica: ce3f521ef3d843beb50dd1217da821d9, State: Running, Role: LEADER
I20260620 11:33:37.958855 3022 external_mini_cluster.cc:1428] Setting key 22487c297af82b60348d1077dcdb4e5d
I20260620 11:33:37.958835 8206 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce3f521ef3d843beb50dd1217da821d9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33677 } }
I20260620 11:33:37.959364 8207 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce3f521ef3d843beb50dd1217da821d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce3f521ef3d843beb50dd1217da821d9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33677 } } }
I20260620 11:33:37.959466 8207 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9 [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:37.959614 8208 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce3f521ef3d843beb50dd1217da821d9. Latest consensus state: current_term: 1 leader_uuid: "ce3f521ef3d843beb50dd1217da821d9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce3f521ef3d843beb50dd1217da821d9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33677 } } }
I20260620 11:33:37.959679 8208 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9 [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:37.959830 8217 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:33:37.960353 8217 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:33:37.962033 8217 catalog_manager.cc:1347] Generated new cluster ID: 972c8301fd5f4c048cbf3f0c8174faea
I20260620 11:33:37.962093 8217 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:33:37.965787 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:37.959381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35526 (local address 127.2.243.190:33677)
0620 11:33:37.959644 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:37.959654 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:37.960305 (+ 651us) server_negotiation.cc:407] Connection header received
0620 11:33:37.960526 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:37.960534 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:37.960605 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:37.960722 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:37.961096 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:37.962408 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:37.963546 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:37.964182 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:37.964308 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:37.965020 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:37.965024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:37.965025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:33:37.965273 (+ 248us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:37.965330 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:33:37.965344 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:37.965415 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:37.965487 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 11:33:37.965565 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":119,"threads_started":1}
I20260620 11:33:37.966753 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:33:37.978091 8217 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:33:37.978793 8217 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:33:37.987217 8217 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce3f521ef3d843beb50dd1217da821d9: Generated new TSK 0
I20260620 11:33:37.987440 8217 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 11:33:38.019559 8233 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 11:33:38.019678 8233 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260620 11:33:38.020308 8232 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:37.998785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33677 (local address 127.0.0.1:35530)
0620 11:33:37.998986 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:37.998999 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:37.999078 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:37.999263 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:37.999264 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:37.999265 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0620 11:33:37.999336 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:37.999339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:37.999965 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:37.999966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:38.000898 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:38.000901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:38.001231 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:38.001232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:38.001296 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:38.001319 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 11:33:38.019851 (+ 18532us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 11:33:38.019958 (+ 107us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.243.190:33677: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":120,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.2.243.190:33677: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 11:33:38.020742 8216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:37.998909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35530 (local address 127.2.243.190:33677)
0620 11:33:37.998962 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:37.998964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:37.999065 (+ 101us) server_negotiation.cc:407] Connection header received
0620 11:33:37.999188 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:37.999188 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:37.999199 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:37.999256 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0620 11:33:37.999402 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:37.999884 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:38.000986 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:38.001161 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:38.001208 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:38.001431 (+ 223us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 11:33:38.019690 (+ 18259us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 11:33:38.020547 (+ 857us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35530: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 11:33:38.020814 8216 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35530: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
I20260620 11:33:38.020826 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 8136
2026-06-20T11:33:38Z chronyd exiting
I20260620 11:33:38.056949 3022 test_util.cc:175] -----------------------------------------------
I20260620 11:33:38.057034 3022 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955192827595-3022-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (252 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 11:33:38.057816 3022 mini_oidc.cc:152] Starting JWKS server
I20260620 11:33:38.058501 3022 webserver.cc:460] Webserver started at https://127.0.0.1:41355/ using document root <none> and password file <none>
I20260620 11:33:38.058542 3022 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 11:33:38.058734 3022 webserver.cc:460] Webserver started at http://127.0.0.1:33929/ using document root <none> and password file <none>
2026-06-20T11:33:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:33:38Z Disabled control of system clock
I20260620 11:33:38.066089 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:40535
--webserver_interface=127.2.243.190
--webserver_port=0
--builtin_ntp_servers=127.2.243.148:35039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:40535
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41355/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955192827595-3022-0/testchainca.cert with env {}
W20260620 11:33:38.146803 8239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:38.147137 8239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:38.147171 8239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:38.148365 8239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:33:38.148428 8239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:38.148452 8239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:38.148483 8239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:33:38.148510 8239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 11:33:38.148540 8239 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 11:33:38.148566 8239 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41355/jwks.json
I20260620 11:33:38.150084 8239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:35039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:40535
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:40535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41355/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955192827595-3022-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:38.150352 8239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:38.150555 8239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:38.155191 8247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:38.155197 8244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:38.155272 8245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:38.155551 8239 server_base.cc:1034] running on GCE node
I20260620 11:33:38.155808 8239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:38.156096 8239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:38.157279 8239 hybrid_clock.cc:648] HybridClock initialized: now 1781955218157256 us; error 43 us; skew 500 ppm
I20260620 11:33:38.158573 8239 webserver.cc:460] Webserver started at http://127.2.243.190:43589/ using document root <none> and password file <none>
I20260620 11:33:38.158829 8239 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:38.158927 8239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:38.159080 8239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:38.160229 8239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "5800184f9a9c45e183a1abc8cad17c44"
format_stamp: "Formatted at 2026-06-20 11:33:38 on dist-test-slave-ctbw"
server_key: "b2d587aa5733ca667868f0cc6c50de41"
server_key_iv: "6ff79dda9d67c85b5afa93d9812843db"
server_key_version: "encryptionkey@0"
I20260620 11:33:38.160586 8239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "5800184f9a9c45e183a1abc8cad17c44"
format_stamp: "Formatted at 2026-06-20 11:33:38 on dist-test-slave-ctbw"
server_key: "b2d587aa5733ca667868f0cc6c50de41"
server_key_iv: "6ff79dda9d67c85b5afa93d9812843db"
server_key_version: "encryptionkey@0"
I20260620 11:33:38.161873 8239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:33:38.162547 8253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:38.162704 8239 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:33:38.162815 8239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "5800184f9a9c45e183a1abc8cad17c44"
format_stamp: "Formatted at 2026-06-20 11:33:38 on dist-test-slave-ctbw"
server_key: "b2d587aa5733ca667868f0cc6c50de41"
server_key_iv: "6ff79dda9d67c85b5afa93d9812843db"
server_key_version: "encryptionkey@0"
I20260620 11:33:38.162907 8239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:38.181350 8239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:38.181658 8239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:38.181814 8239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:38.185837 8239 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:40535
I20260620 11:33:38.185875 8305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:40535 every 8 connection(s)
I20260620 11:33:38.186242 8239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:33:38.187279 8306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:38.189621 8306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44: Bootstrap starting.
I20260620 11:33:38.190162 8306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:38.190320 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 8239
I20260620 11:33:38.190423 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:33:38.190518 8306 log.cc:824] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:38.190634 3022 external_mini_cluster.cc:1428] Setting key 98ffad807d19e04c5242dae6467af46b
I20260620 11:33:38.191416 8306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44: No bootstrap required, opened a new log
I20260620 11:33:38.193851 8306 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5800184f9a9c45e183a1abc8cad17c44" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40535 } }
I20260620 11:33:38.194017 8306 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:38.194047 8306 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5800184f9a9c45e183a1abc8cad17c44, State: Initialized, Role: FOLLOWER
I20260620 11:33:38.194173 8306 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5800184f9a9c45e183a1abc8cad17c44" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40535 } }
I20260620 11:33:38.194267 8306 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:33:38.194310 8306 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:33:38.194351 8306 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:38.194983 8306 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5800184f9a9c45e183a1abc8cad17c44" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40535 } }
I20260620 11:33:38.195101 8306 leader_election.cc:302] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5800184f9a9c45e183a1abc8cad17c44; no voters:
I20260620 11:33:38.195283 8306 leader_election.cc:288] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:33:38.195302 8311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:38.195411 8311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44 [term 1 LEADER]: Becoming Leader. State: Replica: 5800184f9a9c45e183a1abc8cad17c44, State: Running, Role: LEADER
I20260620 11:33:38.195482 8311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5800184f9a9c45e183a1abc8cad17c44" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40535 } }
I20260620 11:33:38.195538 8306 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:38.195794 8311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5800184f9a9c45e183a1abc8cad17c44. Latest consensus state: current_term: 1 leader_uuid: "5800184f9a9c45e183a1abc8cad17c44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5800184f9a9c45e183a1abc8cad17c44" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40535 } } }
I20260620 11:33:38.195811 8313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5800184f9a9c45e183a1abc8cad17c44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5800184f9a9c45e183a1abc8cad17c44" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40535 } } }
I20260620 11:33:38.195860 8311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44 [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:38.195904 8313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44 [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:38.196086 8316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:33:38.196626 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:38.191148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53108 (local address 127.2.243.190:40535)
0620 11:33:38.191335 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:38.191342 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:38.191395 (+ 53us) server_negotiation.cc:407] Connection header received
0620 11:33:38.191871 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:38.191878 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:38.191934 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:38.192044 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:38.192320 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:38.193112 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:38.194284 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:38.194792 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:38.194959 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:38.195651 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:38.195654 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:38.195655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:33:38.195866 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:38.195908 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:33:38.195923 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:38.195990 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:38.196330 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 11:33:38.196408 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260620 11:33:38.196760 8316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:33:38.197588 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:33:38.198527 8316 catalog_manager.cc:1347] Generated new cluster ID: 5925be17ecce4bdf99bcd5ccc58068e4
I20260620 11:33:38.198575 8316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:33:38.214003 8316 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:33:38.214506 8316 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:33:38.218575 8316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5800184f9a9c45e183a1abc8cad17c44: Generated new TSK 0
I20260620 11:33:38.218739 8316 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 11:33:38.247531 8336 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 11:33:38.247635 8336 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 11:33:38.247948 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:38.233093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53118 (local address 127.2.243.190:40535)
0620 11:33:38.233148 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:38.233150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:38.233231 (+ 81us) server_negotiation.cc:407] Connection header received
0620 11:33:38.233304 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:38.233305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:38.233318 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:38.233400 (+ 82us) server_negotiation.cc:226] Negotiated authn=JWT
0620 11:33:38.233535 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:38.234345 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:38.235309 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:38.235618 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:38.235695 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:38.235869 (+ 174us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 11:33:38.247719 (+ 11850us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41355/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 11:33:38.247873 (+ 154us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53118: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41355/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":27}
W20260620 11:33:38.248009 8310 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53118: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41355/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 11:33:38.248093 8335 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:38.233010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40535 (local address 127.0.0.1:53118)
0620 11:33:38.233154 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:38.233164 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:38.233228 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:38.233404 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:38.233404 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:38.233406 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 11:33:38.233463 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:38.233465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:38.234463 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:38.234464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:38.235216 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:38.235218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:38.235698 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:38.235699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:38.235742 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:38.235748 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 11:33:38.247860 (+ 12112us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41355/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 11:33:38.247970 (+ 110us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.243.190:40535: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41355/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":151,"threads_started":1}
I20260620 11:33:38.248399 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 8239
2026-06-20T11:33:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (227 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 11:33:38.285398 3022 mini_oidc.cc:152] Starting JWKS server
I20260620 11:33:38.286039 3022 webserver.cc:460] Webserver started at https://127.0.0.1:42497/ using document root <none> and password file <none>
I20260620 11:33:38.286079 3022 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 11:33:38.286216 3022 webserver.cc:460] Webserver started at http://127.0.0.1:40395/ using document root <none> and password file <none>
2026-06-20T11:33:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:33:38Z Disabled control of system clock
I20260620 11:33:38.293701 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:43827
--webserver_interface=127.2.243.190
--webserver_port=0
--builtin_ntp_servers=127.2.243.148:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:43827
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42497/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 11:33:38.373317 8342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:38.373538 8342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:38.373560 8342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:38.374838 8342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:33:38.374956 8342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:38.374984 8342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:38.374997 8342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:33:38.375010 8342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 11:33:38.375023 8342 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 11:33:38.375034 8342 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42497/jwks.json
I20260620 11:33:38.376381 8342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:43827
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:43827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42497/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:38.376576 8342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:38.376817 8342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:38.381515 8347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:38.381502 8348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:38.381496 8350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:38.381772 8342 server_base.cc:1034] running on GCE node
I20260620 11:33:38.382028 8342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:38.382313 8342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:38.383451 8342 hybrid_clock.cc:648] HybridClock initialized: now 1781955218383438 us; error 32 us; skew 500 ppm
I20260620 11:33:38.384742 8342 webserver.cc:460] Webserver started at http://127.2.243.190:43879/ using document root <none> and password file <none>
I20260620 11:33:38.384980 8342 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:38.385027 8342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:38.385197 8342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:38.386340 8342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "4886632dfd7041d3bc05347cf85997d2"
format_stamp: "Formatted at 2026-06-20 11:33:38 on dist-test-slave-ctbw"
server_key: "86790e634154ff55a1617fb33568bd7f"
server_key_iv: "ae738a566d0160c9b65996caf766e72f"
server_key_version: "encryptionkey@0"
I20260620 11:33:38.386700 8342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "4886632dfd7041d3bc05347cf85997d2"
format_stamp: "Formatted at 2026-06-20 11:33:38 on dist-test-slave-ctbw"
server_key: "86790e634154ff55a1617fb33568bd7f"
server_key_iv: "ae738a566d0160c9b65996caf766e72f"
server_key_version: "encryptionkey@0"
I20260620 11:33:38.388062 8342 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:38.388741 8356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:38.388902 8342 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:33:38.389000 8342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "4886632dfd7041d3bc05347cf85997d2"
format_stamp: "Formatted at 2026-06-20 11:33:38 on dist-test-slave-ctbw"
server_key: "86790e634154ff55a1617fb33568bd7f"
server_key_iv: "ae738a566d0160c9b65996caf766e72f"
server_key_version: "encryptionkey@0"
I20260620 11:33:38.389084 8342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:38.409077 8342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:38.409413 8342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:38.409572 8342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:38.413627 8342 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:43827
I20260620 11:33:38.413666 8408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:43827 every 8 connection(s)
I20260620 11:33:38.414012 8342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:33:38.414678 8409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:38.417259 8409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2: Bootstrap starting.
I20260620 11:33:38.417876 8409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:38.417992 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 8342
I20260620 11:33:38.418118 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:33:38.418210 8409 log.cc:824] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:38.418310 3022 external_mini_cluster.cc:1428] Setting key ac5324496b7ed57f8b4b55991f429755
I20260620 11:33:38.419013 8409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2: No bootstrap required, opened a new log
I20260620 11:33:38.421072 8409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4886632dfd7041d3bc05347cf85997d2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43827 } }
I20260620 11:33:38.421231 8409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:38.421280 8409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4886632dfd7041d3bc05347cf85997d2, State: Initialized, Role: FOLLOWER
I20260620 11:33:38.421411 8409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4886632dfd7041d3bc05347cf85997d2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43827 } }
I20260620 11:33:38.421492 8409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:33:38.421536 8409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:33:38.421583 8409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:38.422147 8409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4886632dfd7041d3bc05347cf85997d2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43827 } }
I20260620 11:33:38.422256 8409 leader_election.cc:302] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4886632dfd7041d3bc05347cf85997d2; no voters:
I20260620 11:33:38.422426 8409 leader_election.cc:288] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:33:38.422459 8414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:38.422619 8414 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2 [term 1 LEADER]: Becoming Leader. State: Replica: 4886632dfd7041d3bc05347cf85997d2, State: Running, Role: LEADER
I20260620 11:33:38.422683 8409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:38.422710 8414 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4886632dfd7041d3bc05347cf85997d2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43827 } }
I20260620 11:33:38.423043 8416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4886632dfd7041d3bc05347cf85997d2. Latest consensus state: current_term: 1 leader_uuid: "4886632dfd7041d3bc05347cf85997d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4886632dfd7041d3bc05347cf85997d2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43827 } } }
I20260620 11:33:38.423175 8416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2 [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:38.423418 8421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:33:38.423048 8415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4886632dfd7041d3bc05347cf85997d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4886632dfd7041d3bc05347cf85997d2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43827 } } }
I20260620 11:33:38.423578 8415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2 [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:38.424006 8421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:33:38.424270 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:38.418762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51982 (local address 127.2.243.190:43827)
0620 11:33:38.418995 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:38.419005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:38.419037 (+ 32us) server_negotiation.cc:407] Connection header received
0620 11:33:38.419496 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:38.419503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:38.419557 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:38.419707 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:38.419919 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:38.420766 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:38.421692 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:38.422092 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:38.422178 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:38.423092 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:38.423097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:38.423097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:33:38.423341 (+ 244us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:38.423386 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:33:38.423405 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:38.423828 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:38.423867 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 11:33:38.423962 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":104,"threads_started":1}
I20260620 11:33:38.425325 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:33:38.427449 8421 catalog_manager.cc:1347] Generated new cluster ID: adbff74ec8004f4382494ba3f7d0c451
I20260620 11:33:38.427513 8421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:33:38.433781 8421 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:33:38.434484 8421 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:33:38.453671 8421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4886632dfd7041d3bc05347cf85997d2: Generated new TSK 0
I20260620 11:33:38.453907 8421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:33:38.470454 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:38.466813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51990 (local address 127.2.243.190:43827)
0620 11:33:38.466889 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:38.466891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:38.467030 (+ 139us) server_negotiation.cc:407] Connection header received
0620 11:33:38.467081 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:38.467081 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:38.467094 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:38.467169 (+ 75us) server_negotiation.cc:226] Negotiated authn=JWT
0620 11:33:38.467266 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:38.468053 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:38.468784 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:38.469106 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:38.469175 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:38.469322 (+ 147us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 11:33:38.470231 (+ 909us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42497/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 11:33:38.470373 (+ 142us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51990: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42497/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 11:33:38.470513 8413 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51990: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42497/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 11:33:38.472434 8438 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:38.466719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43827 (local address 127.0.0.1:51990)
0620 11:33:38.466958 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:38.466969 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:38.467027 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:38.467180 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:38.467181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:38.467182 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0620 11:33:38.467228 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:38.467230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:38.468131 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:38.468131 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:38.468709 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:38.468710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:38.469182 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:38.469183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:38.469222 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:38.469226 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 11:33:38.470334 (+ 1108us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42497/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 11:33:38.472340 (+ 2006us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.243.190:43827: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42497/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":1822,"thread_start_us":148,"threads_started":1}
I20260620 11:33:38.472851 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 8342
E20260620 11:33:38.481441 8439 webserver.cc:533] Webserver: sslize failed:
E20260620 11:33:38.481508 8439 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T11:33:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (224 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T11:33:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:33:38Z Disabled control of system clock
I20260620 11:33:38.517486 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:45925
--webserver_interface=127.2.243.190
--webserver_port=0
--builtin_ntp_servers=127.2.243.148:39303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:45925
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955192827595-3022-0/testchainca.cert with env {}
W20260620 11:33:38.599895 8443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:38.600095 8443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:38.600117 8443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:38.601255 8443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:33:38.601296 8443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:38.601310 8443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:38.601323 8443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:33:38.601336 8443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 11:33:38.601347 8443 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 11:33:38.601359 8443 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 11:33:38.602741 8443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:39303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:45925
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:45925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955192827595-3022-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8443
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:38.602983 8443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:38.603156 8443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:38.607933 8448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:38.607923 8451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:38.607975 8443 server_base.cc:1034] running on GCE node
W20260620 11:33:38.607924 8449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:38.608356 8443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:38.608654 8443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:38.609860 8443 hybrid_clock.cc:648] HybridClock initialized: now 1781955218609851 us; error 52 us; skew 500 ppm
I20260620 11:33:38.611266 8443 webserver.cc:460] Webserver started at http://127.2.243.190:46107/ using document root <none> and password file <none>
I20260620 11:33:38.611521 8443 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:38.611588 8443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:38.611732 8443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:38.612807 8443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "bd2156497bef4ed3b35916dbd48a1b3d"
format_stamp: "Formatted at 2026-06-20 11:33:38 on dist-test-slave-ctbw"
server_key: "b9e1fe2d1b8e3b96ff30e93001b78635"
server_key_iv: "657f4747d9002c0225da17cafd056abc"
server_key_version: "encryptionkey@0"
I20260620 11:33:38.613155 8443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "bd2156497bef4ed3b35916dbd48a1b3d"
format_stamp: "Formatted at 2026-06-20 11:33:38 on dist-test-slave-ctbw"
server_key: "b9e1fe2d1b8e3b96ff30e93001b78635"
server_key_iv: "657f4747d9002c0225da17cafd056abc"
server_key_version: "encryptionkey@0"
I20260620 11:33:38.614440 8443 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:33:38.615231 8457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:38.615391 8443 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:38.615500 8443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "bd2156497bef4ed3b35916dbd48a1b3d"
format_stamp: "Formatted at 2026-06-20 11:33:38 on dist-test-slave-ctbw"
server_key: "b9e1fe2d1b8e3b96ff30e93001b78635"
server_key_iv: "657f4747d9002c0225da17cafd056abc"
server_key_version: "encryptionkey@0"
I20260620 11:33:38.615573 8443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:38.629276 8443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:38.629587 8443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:38.629743 8443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:38.633579 8443 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:45925
I20260620 11:33:38.633630 8509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:45925 every 8 connection(s)
I20260620 11:33:38.633972 8443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:33:38.634963 8510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:38.637145 8510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d: Bootstrap starting.
I20260620 11:33:38.637707 8510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:38.637995 8510 log.cc:824] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:38.638711 8510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d: No bootstrap required, opened a new log
I20260620 11:33:38.640328 8510 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2156497bef4ed3b35916dbd48a1b3d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 45925 } }
I20260620 11:33:38.640470 8510 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:38.640493 8510 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd2156497bef4ed3b35916dbd48a1b3d, State: Initialized, Role: FOLLOWER
I20260620 11:33:38.640655 8510 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2156497bef4ed3b35916dbd48a1b3d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 45925 } }
I20260620 11:33:38.640719 8510 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:33:38.640741 8510 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:33:38.640769 8510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:38.641361 8510 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2156497bef4ed3b35916dbd48a1b3d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 45925 } }
I20260620 11:33:38.641474 8510 leader_election.cc:302] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd2156497bef4ed3b35916dbd48a1b3d; no voters:
I20260620 11:33:38.641650 8510 leader_election.cc:288] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:33:38.641662 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 8443
I20260620 11:33:38.641709 8513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:38.641767 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:33:38.641862 8513 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d [term 1 LEADER]: Becoming Leader. State: Replica: bd2156497bef4ed3b35916dbd48a1b3d, State: Running, Role: LEADER
I20260620 11:33:38.641974 3022 external_mini_cluster.cc:1428] Setting key 93cbd40731a411bcd51ac31a2b9dac1f
I20260620 11:33:38.641963 8510 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:38.641973 8513 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2156497bef4ed3b35916dbd48a1b3d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 45925 } }
I20260620 11:33:38.642437 8513 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd2156497bef4ed3b35916dbd48a1b3d. Latest consensus state: current_term: 1 leader_uuid: "bd2156497bef4ed3b35916dbd48a1b3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2156497bef4ed3b35916dbd48a1b3d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 45925 } } }
I20260620 11:33:38.642513 8513 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:38.642680 8514 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd2156497bef4ed3b35916dbd48a1b3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd2156497bef4ed3b35916dbd48a1b3d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 45925 } } }
I20260620 11:33:38.642748 8514 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:38.642882 8521 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:33:38.643666 8521 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:33:38.645387 8521 catalog_manager.cc:1347] Generated new cluster ID: 5e4a2149d6ef41fa97b84eb6e6f4791a
I20260620 11:33:38.645447 8521 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:33:38.648417 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:38.642622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36096 (local address 127.2.243.190:45925)
0620 11:33:38.642832 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:38.642842 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:38.643049 (+ 207us) server_negotiation.cc:407] Connection header received
0620 11:33:38.643282 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:38.643289 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:38.643390 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:38.643516 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:38.643911 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:38.645141 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:38.646243 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:38.646889 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:38.647042 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:38.647717 (+ 675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:38.647720 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:38.647721 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:33:38.647925 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:38.647976 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:33:38.647992 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:38.648060 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:38.648130 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 11:33:38.648205 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":70,"threads_started":1}
I20260620 11:33:38.651933 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:33:38.657015 8521 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:33:38.657485 8521 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:33:38.661603 8521 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd2156497bef4ed3b35916dbd48a1b3d: Generated new TSK 0
I20260620 11:33:38.661762 8521 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:33:38.731156 8524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:38.668028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36112 (local address 127.2.243.190:45925)
0620 11:33:38.668069 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:38.668071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:38.668157 (+ 86us) server_negotiation.cc:407] Connection header received
0620 11:33:38.668194 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:38.668195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:38.668207 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:38.668278 (+ 71us) server_negotiation.cc:226] Negotiated authn=JWT
0620 11:33:38.668384 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:38.669454 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:38.670216 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:38.670537 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:38.670609 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:38.670833 (+ 224us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 11:33:38.730827 (+ 59994us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 11:33:38.731076 (+ 249us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36112: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 11:33:38.731240 8524 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36112: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 11:33:38.731338 8539 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:38.667943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:45925 (local address 127.0.0.1:36112)
0620 11:33:38.668083 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:38.668093 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:38.668151 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:38.668291 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:38.668292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:38.668293 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0620 11:33:38.668342 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:38.668343 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:38.669556 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:38.669557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:38.670137 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:38.670139 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:38.670639 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:38.670640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:38.670690 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:38.670695 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 11:33:38.731037 (+ 60342us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 11:33:38.731178 (+ 141us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.2.243.190:45925: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":94,"threads_started":1}
I20260620 11:33:38.731706 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 8443
2026-06-20T11:33:38Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (239 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (87 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:33:38 dist-test-slave-ctbw krb5kdc[8551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:33:38 dist-test-slave-ctbw krb5kdc[8551](info): set up 2 sockets
Jun 20 11:33:38 dist-test-slave-ctbw krb5kdc[8551](info): commencing operation
krb5kdc: starting...
W20260620 11:33:39.983363 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:33:40 dist-test-slave-ctbw krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:33:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:33:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:33:40.102511 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:39331
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:44061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:39331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:40.181770 8564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:40.182005 8564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:40.182044 8564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:40.183214 8564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:33:40.183282 8564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:40.183306 8564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:40.183336 8564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:33:40.183367 8564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:33:40.184760 8564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:44061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:39331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:39331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:40.185034 8564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:40.185254 8564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:40.189987 8569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:40.190093 8572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:40.190038 8570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:40.190164 8564 server_base.cc:1034] running on GCE node
I20260620 11:33:40.190448 8564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:40.190739 8564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:40.191905 8564 hybrid_clock.cc:648] HybridClock initialized: now 1781955220191905 us; error 47 us; skew 500 ppm
Jun 20 11:33:40 dist-test-slave-ctbw krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955220, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:40.194337 8564 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:33:40.194823 8564 webserver.cc:460] Webserver started at http://127.2.243.190:39409/ using document root <none> and password file <none>
I20260620 11:33:40.195077 8564 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:40.195154 8564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:40.195302 8564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:40.196305 8564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "d382baa3cad148a99ed80e033def7c2d"
format_stamp: "Formatted at 2026-06-20 11:33:40 on dist-test-slave-ctbw"
server_key: "485e7c7982a97c71bb19c77a6b12f177"
server_key_iv: "4af662a251f69c8ce7fadff5be60add3"
server_key_version: "encryptionkey@0"
I20260620 11:33:40.196673 8564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "d382baa3cad148a99ed80e033def7c2d"
format_stamp: "Formatted at 2026-06-20 11:33:40 on dist-test-slave-ctbw"
server_key: "485e7c7982a97c71bb19c77a6b12f177"
server_key_iv: "4af662a251f69c8ce7fadff5be60add3"
server_key_version: "encryptionkey@0"
I20260620 11:33:40.198036 8564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:33:40.198799 8579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:40.199085 8564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:40.199174 8564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "d382baa3cad148a99ed80e033def7c2d"
format_stamp: "Formatted at 2026-06-20 11:33:40 on dist-test-slave-ctbw"
server_key: "485e7c7982a97c71bb19c77a6b12f177"
server_key_iv: "4af662a251f69c8ce7fadff5be60add3"
server_key_version: "encryptionkey@0"
I20260620 11:33:40.199227 8564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:40.226198 8564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:40.228582 8564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:40.228694 8564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:40.232270 8564 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:39331
I20260620 11:33:40.232326 8631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:39331 every 8 connection(s)
I20260620 11:33:40.232667 8564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:33:40.233530 8632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:40.235733 8632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d: Bootstrap starting.
I20260620 11:33:40.236289 8632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:40.236579 8632 log.cc:824] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:40.237243 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 8564
I20260620 11:33:40.237337 8632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d: No bootstrap required, opened a new log
I20260620 11:33:40.237334 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:33:40.237576 3022 external_mini_cluster.cc:1428] Setting key 62745653a883565b9133ed504138db5d
I20260620 11:33:40.239223 8632 raft_consensus.cc:348] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d382baa3cad148a99ed80e033def7c2d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39331 } }
I20260620 11:33:40.239372 8632 raft_consensus.cc:374] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:40.239434 8632 raft_consensus.cc:729] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d382baa3cad148a99ed80e033def7c2d, State: Initialized, Role: FOLLOWER
I20260620 11:33:40.239617 8632 consensus_queue.cc:260] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d382baa3cad148a99ed80e033def7c2d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39331 } }
I20260620 11:33:40.239709 8632 raft_consensus.cc:388] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:33:40.239748 8632 raft_consensus.cc:482] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:33:40.239797 8632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:40.240645 8632 raft_consensus.cc:504] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d382baa3cad148a99ed80e033def7c2d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39331 } }
I20260620 11:33:40.240761 8632 leader_election.cc:302] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d382baa3cad148a99ed80e033def7c2d; no voters:
I20260620 11:33:40.240960 8632 leader_election.cc:288] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:33:40.240998 8637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:40.241098 8637 raft_consensus.cc:686] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d [term 1 LEADER]: Becoming Leader. State: Replica: d382baa3cad148a99ed80e033def7c2d, State: Running, Role: LEADER
I20260620 11:33:40.241225 8632 sys_catalog.cc:564] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:40.241240 8637 consensus_queue.cc:237] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d382baa3cad148a99ed80e033def7c2d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39331 } }
I20260620 11:33:40.241560 8637 sys_catalog.cc:455] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d382baa3cad148a99ed80e033def7c2d. Latest consensus state: current_term: 1 leader_uuid: "d382baa3cad148a99ed80e033def7c2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d382baa3cad148a99ed80e033def7c2d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39331 } } }
I20260620 11:33:40.241633 8637 sys_catalog.cc:458] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:40.241855 8642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:33:40.242348 8642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:33:40.242744 8638 sys_catalog.cc:455] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d382baa3cad148a99ed80e033def7c2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d382baa3cad148a99ed80e033def7c2d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39331 } } }
I20260620 11:33:40.242827 8638 sys_catalog.cc:458] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:40.245136 8642 catalog_manager.cc:1347] Generated new cluster ID: 62e11bee00594e0dbb40b7cc4373e3d2
I20260620 11:33:40.245193 8642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:33:40 dist-test-slave-ctbw krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:40.249416 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:40.238097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53636 (local address 127.2.243.190:39331)
0620 11:33:40.238286 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:40.238295 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:40.238330 (+ 35us) server_negotiation.cc:407] Connection header received
0620 11:33:40.239219 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:40.239226 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:40.239282 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:40.239436 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:40.240229 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:40.241419 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:40.242472 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:40.243195 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:40.243342 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:40.246058 (+ 2716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:40.246061 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:40.246062 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:40.246073 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:40.247695 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:40.248059 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:40.248065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:40.248066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:40.248162 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:40.248370 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:40.248371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:40.248371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:40.248600 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:40.249139 (+ 539us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:40.249192 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 11:33:40.249286 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
I20260620 11:33:40.253206 8642 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:33:40.253907 8642 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:33:40.258950 8642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d382baa3cad148a99ed80e033def7c2d: Generated new TSK 0
I20260620 11:33:40.259110 8642 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
I20260620 11:33:40.289506 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:39331
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:44061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:40.371886 8658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:40.372102 8658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:40.372125 8658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:40.373319 8658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:40.373366 8658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:40.373394 8658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:33:40.375079 8658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:44061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:39331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:40.375337 8658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:40.375594 8658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:40.380738 8663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:40.380708 8664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:40.380798 8658 server_base.cc:1034] running on GCE node
W20260620 11:33:40.380707 8666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:40.381215 8658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:40.381417 8658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:40.382558 8658 hybrid_clock.cc:648] HybridClock initialized: now 1781955220382545 us; error 37 us; skew 500 ppm
Jun 20 11:33:40 dist-test-slave-ctbw krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955220, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:40.385236 8658 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:33:40.385617 8658 webserver.cc:460] Webserver started at http://127.2.243.129:42187/ using document root <none> and password file <none>
I20260620 11:33:40.385807 8658 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:40.385847 8658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:40.385931 8658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:40.386972 8658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "648b20577535420db55870c975b1ba62"
format_stamp: "Formatted at 2026-06-20 11:33:40 on dist-test-slave-ctbw"
server_key: "0e4be6c48f6b6cf90ab9547692e602f1"
server_key_iv: "f3a5d92d8cd4790518da4ea0ef66fe2e"
server_key_version: "encryptionkey@0"
I20260620 11:33:40.387274 8658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "648b20577535420db55870c975b1ba62"
format_stamp: "Formatted at 2026-06-20 11:33:40 on dist-test-slave-ctbw"
server_key: "0e4be6c48f6b6cf90ab9547692e602f1"
server_key_iv: "f3a5d92d8cd4790518da4ea0ef66fe2e"
server_key_version: "encryptionkey@0"
I20260620 11:33:40.388501 8658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:40.389119 8673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:40.389286 8658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:40.389387 8658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "648b20577535420db55870c975b1ba62"
format_stamp: "Formatted at 2026-06-20 11:33:40 on dist-test-slave-ctbw"
server_key: "0e4be6c48f6b6cf90ab9547692e602f1"
server_key_iv: "f3a5d92d8cd4790518da4ea0ef66fe2e"
server_key_version: "encryptionkey@0"
I20260620 11:33:40.389457 8658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:40.404770 8658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:40.407157 8658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:40.407366 8658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:40.407615 8658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:40.407919 8658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:40.407971 8658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:40.408005 8658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:40.408051 8658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:40.413723 8658 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:45235
I20260620 11:33:40.413774 8786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:45235 every 8 connection(s)
I20260620 11:33:40.414119 8658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:33:40.414448 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 8658
I20260620 11:33:40.414563 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:33:40.414772 3022 external_mini_cluster.cc:1428] Setting key 2461cceea54146d320937e5cb8cc28db
Jun 20 11:33:40 dist-test-slave-ctbw krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955220, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.243.130@KRBTEST.COM; defaulting to no policy
I20260620 11:33:40.424463 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:40.414930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:49799 (local address 127.2.243.190:39331)
0620 11:33:40.414986 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:40.414989 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:40.415202 (+ 213us) server_negotiation.cc:407] Connection header received
0620 11:33:40.416261 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:40.416262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:40.416277 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:40.416341 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:40.417151 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:40.417640 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:40.418657 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:40.418932 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:40.418991 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:40.421606 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:40.421607 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:40.421608 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:40.421615 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:40.423221 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:40.423568 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:40.423569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:40.423569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:40.423618 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:40.423861 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:40.423862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:40.423862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:40.423968 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:40.424059 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:40.424324 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 11:33:40.424398 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 11:33:40.424621 8789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:40.414901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:39331 (local address 127.2.243.129:49799)
0620 11:33:40.415062 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:40.415104 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:40.416151 (+ 1047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:40.416409 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:40.416413 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:40.416760 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:40.417055 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:40.417061 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:40.417741 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:40.417742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:40.418559 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:40.418561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:40.419047 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:40.419048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:40.419169 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:40.419694 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:40.419702 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:40.421485 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:40.423294 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:40.423299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:40.423301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:40.423504 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:40.423693 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:40.423694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:40.423694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:40.423813 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:40.424061 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:40.424065 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:40.424187 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 11:33:40.424271 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 11:33:40.424389 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260620 11:33:40.425164 8787 heartbeater.cc:347] Connected to a master server at 127.2.243.190:39331
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.130@KRBTEST.COM" created.
I20260620 11:33:40.425269 8787 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:40.425437 8787 heartbeater.cc:511] Master 127.2.243.190:39331 requested a full tablet report, sending...
I20260620 11:33:40.426025 8596 ts_manager.cc:194] Registered new tserver with Master: 648b20577535420db55870c975b1ba62 (127.2.243.129:45235)
I20260620 11:33:40.426666 8596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:49799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
I20260620 11:33:40.438769 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:39331
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:44061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:40.518482 8792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:40.518688 8792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:40.518711 8792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:40.519804 8792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:40.519837 8792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:40.519862 8792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:33:40.521210 8792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:44061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:39331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:40.521421 8792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:40.521586 8792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:40.526721 8798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:40.526695 8800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:40.526695 8797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:40.526969 8792 server_base.cc:1034] running on GCE node
I20260620 11:33:40.527217 8792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:40.527472 8792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:40.528632 8792 hybrid_clock.cc:648] HybridClock initialized: now 1781955220528615 us; error 51 us; skew 500 ppm
Jun 20 11:33:40 dist-test-slave-ctbw krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955220, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:40.531217 8792 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:33:40.531634 8792 webserver.cc:460] Webserver started at http://127.2.243.130:38763/ using document root <none> and password file <none>
I20260620 11:33:40.531854 8792 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:40.531919 8792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:40.532061 8792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:40.533140 8792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "1cd4caf39be045b1b182bd7c80e5e9b5"
format_stamp: "Formatted at 2026-06-20 11:33:40 on dist-test-slave-ctbw"
server_key: "2bab46d10ba305eb9d2dc074a0e851e6"
server_key_iv: "1d47491c07819554cb4da970a7b49d9e"
server_key_version: "encryptionkey@0"
I20260620 11:33:40.533492 8792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "1cd4caf39be045b1b182bd7c80e5e9b5"
format_stamp: "Formatted at 2026-06-20 11:33:40 on dist-test-slave-ctbw"
server_key: "2bab46d10ba305eb9d2dc074a0e851e6"
server_key_iv: "1d47491c07819554cb4da970a7b49d9e"
server_key_version: "encryptionkey@0"
I20260620 11:33:40.534821 8792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 11:33:40.535615 8807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:40.535781 8792 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:33:40.535877 8792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "1cd4caf39be045b1b182bd7c80e5e9b5"
format_stamp: "Formatted at 2026-06-20 11:33:40 on dist-test-slave-ctbw"
server_key: "2bab46d10ba305eb9d2dc074a0e851e6"
server_key_iv: "1d47491c07819554cb4da970a7b49d9e"
server_key_version: "encryptionkey@0"
I20260620 11:33:40.535954 8792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:40.543895 8792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:40.546159 8792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:40.546305 8792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:40.546531 8792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:40.546814 8792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:40.546864 8792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:40.546922 8792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:40.546960 8792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:40.552712 8792 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:36219
I20260620 11:33:40.552757 8920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:36219 every 8 connection(s)
I20260620 11:33:40.553084 8792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:33:40.553431 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 8792
I20260620 11:33:40.553545 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:33:40.553740 3022 external_mini_cluster.cc:1428] Setting key 01816cfb21892fc1b707ea5e8ac27bcc
Jun 20 11:33:40 dist-test-slave-ctbw krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955220, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.243.131@KRBTEST.COM; defaulting to no policy
I20260620 11:33:40.563591 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:40.553859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:45053 (local address 127.2.243.190:39331)
0620 11:33:40.553913 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:40.553915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:40.554120 (+ 205us) server_negotiation.cc:407] Connection header received
0620 11:33:40.555385 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:40.555386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:40.555396 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:40.555439 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:40.556277 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:40.557047 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:40.558130 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:40.558457 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:40.558537 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:40.560701 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:40.560703 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:40.560704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:40.560712 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:40.562273 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:40.562706 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:40.562708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:40.562708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:40.562768 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:40.563029 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:40.563030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:40.563030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:40.563141 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:40.563192 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:40.563456 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 11:33:40.563527 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:33:40.563867 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:40.553862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:39331 (local address 127.2.243.130:45053)
0620 11:33:40.554012 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:40.554048 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:40.555271 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:40.555513 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:40.555517 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:40.555919 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:40.556185 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:40.556191 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:40.557189 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:40.557189 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:40.558018 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:40.558021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:40.558538 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:40.558539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:40.558652 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:40.559246 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:40.559255 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:40.560594 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:40.562356 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:40.562361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:40.562363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:40.562635 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:40.562835 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:40.562836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:40.562836 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:40.562984 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:40.563194 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:40.563198 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:40.563310 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 11:33:40.563416 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 11:33:40.563604 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260620 11:33:40.564443 8921 heartbeater.cc:347] Connected to a master server at 127.2.243.190:39331
I20260620 11:33:40.564555 8921 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:40.564774 8921 heartbeater.cc:511] Master 127.2.243.190:39331 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.131@KRBTEST.COM" created.
I20260620 11:33:40.565158 8596 ts_manager.cc:194] Registered new tserver with Master: 1cd4caf39be045b1b182bd7c80e5e9b5 (127.2.243.130:36219)
I20260620 11:33:40.565526 8596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:45053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
I20260620 11:33:40.579852 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:39331
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:44061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:40.659312 8926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:40.659503 8926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:40.659525 8926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:40.660651 8926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:40.660692 8926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:40.660718 8926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:33:40.662150 8926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:44061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:39331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:40.662389 8926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:40.662611 8926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:40.667622 8932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:40.667723 8926 server_base.cc:1034] running on GCE node
W20260620 11:33:40.667637 8934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:40.667631 8931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:40.668056 8926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:40.668315 8926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:40.669448 8926 hybrid_clock.cc:648] HybridClock initialized: now 1781955220669437 us; error 31 us; skew 500 ppm
Jun 20 11:33:40 dist-test-slave-ctbw krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955220, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:40.671903 8926 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:33:40.672364 8926 webserver.cc:460] Webserver started at http://127.2.243.131:34485/ using document root <none> and password file <none>
I20260620 11:33:40.672600 8926 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:40.672643 8926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:40.672772 8926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:40.673776 8926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "3788c9f564b540e09c2e4da4e5ceadfe"
format_stamp: "Formatted at 2026-06-20 11:33:40 on dist-test-slave-ctbw"
server_key: "6bcf9a1490ad456ba81c7f035aa87330"
server_key_iv: "6fad4005c486f4b339c510b12b054b4a"
server_key_version: "encryptionkey@0"
I20260620 11:33:40.674111 8926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "3788c9f564b540e09c2e4da4e5ceadfe"
format_stamp: "Formatted at 2026-06-20 11:33:40 on dist-test-slave-ctbw"
server_key: "6bcf9a1490ad456ba81c7f035aa87330"
server_key_iv: "6fad4005c486f4b339c510b12b054b4a"
server_key_version: "encryptionkey@0"
I20260620 11:33:40.675519 8926 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:33:40.676249 8941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:40.676435 8926 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:40.676527 8926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "3788c9f564b540e09c2e4da4e5ceadfe"
format_stamp: "Formatted at 2026-06-20 11:33:40 on dist-test-slave-ctbw"
server_key: "6bcf9a1490ad456ba81c7f035aa87330"
server_key_iv: "6fad4005c486f4b339c510b12b054b4a"
server_key_version: "encryptionkey@0"
I20260620 11:33:40.676580 8926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:40.684458 8926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:40.686362 8926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:40.686484 8926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:40.686666 8926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:40.687009 8926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:40.687053 8926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:40.687077 8926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:40.687093 8926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:40.693498 8926 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:38427
I20260620 11:33:40.693606 9054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:38427 every 8 connection(s)
I20260620 11:33:40.693907 8926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:33:40.694416 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 8926
I20260620 11:33:40.694547 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:33:40.694787 3022 external_mini_cluster.cc:1428] Setting key 41e5b03eba876f41823655297082591a
Jun 20 11:33:40 dist-test-slave-ctbw krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955220, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:40.704097 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:40.694755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:43763 (local address 127.2.243.190:39331)
0620 11:33:40.694832 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:40.694834 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:40.695236 (+ 402us) server_negotiation.cc:407] Connection header received
0620 11:33:40.696346 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:40.696347 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:40.696358 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:40.696403 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:40.697028 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:40.697529 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:40.698545 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:40.698745 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:40.698831 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:40.701263 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:40.701265 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:40.701265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:40.701275 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:40.702781 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:40.703223 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:40.703224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:40.703225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:40.703280 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:40.703533 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:40.703533 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:40.703533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:40.703637 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:40.703686 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:40.703922 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 11:33:40.703984 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 11:33:40.704228 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:40.694754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:39331 (local address 127.2.243.131:43763)
0620 11:33:40.695105 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:40.695154 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:40.696243 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:40.696474 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:40.696477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:40.696735 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:40.696918 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:40.696922 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:40.697619 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:40.697620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:40.698437 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:40.698439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:40.698847 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:40.698848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:40.699109 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:40.699580 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:40.699589 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:40.701119 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:40.702850 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:40.702870 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:40.702890 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:40.703139 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:40.703365 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:40.703366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:40.703366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:40.703464 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:40.703691 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:40.703695 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:40.703807 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 11:33:40.703884 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 11:33:40.703999 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":83,"threads_started":1}
I20260620 11:33:40.704732 9055 heartbeater.cc:347] Connected to a master server at 127.2.243.190:39331
I20260620 11:33:40.704838 9055 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:40.705013 9055 heartbeater.cc:511] Master 127.2.243.190:39331 requested a full tablet report, sending...
I20260620 11:33:40.705382 8596 ts_manager.cc:194] Registered new tserver with Master: 3788c9f564b540e09c2e4da4e5ceadfe (127.2.243.131:38427)
I20260620 11:33:40.705747 8596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:43763
I20260620 11:33:40.706714 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/test-admin.keytab.
Jun 20 11:33:40 dist-test-slave-ctbw krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:40.723577 3022 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 11:33:40.725318 9062 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:40.724533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43362 (local address 127.2.243.129:45235)
0620 11:33:40.724704 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:40.724709 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:40.724719 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:33:40.724745 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:40.724746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:40.724760 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:40.724828 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:40.725230 (+ 402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43362: BlockingRecv error: recv got EOF from 127.0.0.1:43362 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
W20260620 11:33:40.725414 3022 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/krb5cc)
Jun 20 11:33:40 dist-test-slave-ctbw krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:40.726419 3022 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 11:33:40 dist-test-slave-ctbw krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:40.727339 3022 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 11:33:40 dist-test-slave-ctbw krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.129@KRBTEST.COM
I20260620 11:33:40.735193 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:40.727906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43364 (local address 127.2.243.129:45235)
0620 11:33:40.727963 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:40.727964 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:40.727980 (+ 16us) server_negotiation.cc:407] Connection header received
0620 11:33:40.728002 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:40.728002 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:40.728012 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:40.728070 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:40.728432 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:40.729185 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:40.729964 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:40.730292 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:40.730348 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:40.732488 (+ 2140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:40.732494 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:40.732494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:40.732500 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:40.733995 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:40.734327 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:40.734329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:40.734329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:40.734381 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:40.734622 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:40.734624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:40.734624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:40.734775 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:40.734963 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:40.735046 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 11:33:40.735112 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260620 11:33:40.737203 9062 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:40.736609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43380 (local address 127.2.243.129:45235)
0620 11:33:40.736665 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:40.736667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:40.736742 (+ 75us) server_negotiation.cc:407] Connection header received
0620 11:33:40.736814 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:40.736815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:40.736826 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:40.736875 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:40.737156 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43380: BlockingRecv error: recv got EOF from 127.0.0.1:43380 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260620 11:33:40.737330 3022 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/krb5cc' of type FILE
Jun 20 11:33:40 dist-test-slave-ctbw krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:40.738308 3022 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 11:33:40 dist-test-slave-ctbw krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:40.739408 3022 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 11:33:40 dist-test-slave-ctbw krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.129@KRBTEST.COM
I20260620 11:33:40.746842 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:40.739958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.2.243.129:45235)
0620 11:33:40.740016 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:40.740018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:40.740047 (+ 29us) server_negotiation.cc:407] Connection header received
0620 11:33:40.740100 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:40.740101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:40.740109 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:40.740156 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:40.740476 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:40.741012 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:40.741859 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:40.742162 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:40.742228 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:40.744379 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:40.744381 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:40.744381 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:40.744387 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:40.745828 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:40.746133 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:40.746134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:40.746134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:40.746184 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:40.746390 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:40.746390 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:40.746390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:40.746474 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:40.746542 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:40.746652 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 11:33:40.746708 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 11:33:40.748373 9062 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:40.747798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43398 (local address 127.2.243.129:45235)
0620 11:33:40.747829 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:40.747831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:40.747884 (+ 53us) server_negotiation.cc:407] Connection header received
0620 11:33:40.747921 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:40.747921 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:40.747928 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:40.747973 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:40.748329 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43398: BlockingRecv error: recv got EOF from 127.0.0.1:43398 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260620 11:33:40.748450 3022 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955192827595-3022-0/krb5kdc/krb5cc' of type FILE
Jun 20 11:33:40 dist-test-slave-ctbw krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:40.749482 3022 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 11:33:40 dist-test-slave-ctbw krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:40.750449 3022 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 11:33:40 dist-test-slave-ctbw krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.129@KRBTEST.COM
I20260620 11:33:40.758370 9062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:40.750975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43414 (local address 127.2.243.129:45235)
0620 11:33:40.751010 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:40.751012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:40.751138 (+ 126us) server_negotiation.cc:407] Connection header received
0620 11:33:40.751210 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:40.751211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:40.751220 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:40.751289 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:40.751682 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:40.752178 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:40.752906 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:40.753097 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:40.753172 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:40.755311 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:40.755312 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:40.755312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:40.755319 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:40.757005 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:40.757418 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:40.757420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:40.757420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:40.757484 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:40.757749 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:40.757750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:40.757750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:40.757898 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:40.758070 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:40.758175 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 11:33:40.758253 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 11:33:40.759266 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 8658
I20260620 11:33:40.764083 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 8792
I20260620 11:33:40.769140 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 8926
I20260620 11:33:40.773599 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 8564
2026-06-20T11:33:40Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1869 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:33:40 dist-test-slave-ctbw krb5kdc[9075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:33:40 dist-test-slave-ctbw krb5kdc[9075](info): set up 2 sockets
Jun 20 11:33:40 dist-test-slave-ctbw krb5kdc[9075](info): commencing operation
krb5kdc: starting...
W20260620 11:33:41.815582 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:33:41 dist-test-slave-ctbw krb5kdc[9075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:33:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:33:41Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.190.keytab.
Entry for principal oryx/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.190.keytab.
I20260620 11:33:41.934410 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:41191
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.190.keytab
--principal=oryx/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:34177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:41191
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:42.014187 9088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:42.014434 9088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:42.014472 9088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:42.015688 9088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:33:42.015755 9088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:42.015781 9088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:42.015815 9088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:33:42.015838 9088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:33:42.017304 9088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:34177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:41191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.190.keytab
--principal=oryx/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:41191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:42.017575 9088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:42.017800 9088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:42.022642 9093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:42.022814 9088 server_base.cc:1034] running on GCE node
W20260620 11:33:42.022670 9096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:42.022642 9094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:42.023105 9088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:42.023422 9088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:42.024608 9088 hybrid_clock.cc:648] HybridClock initialized: now 1781955222024590 us; error 39 us; skew 500 ppm
Jun 20 11:33:42 dist-test-slave-ctbw krb5kdc[9075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955222, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:42.029842 9088 init.cc:377] Logged in from keytab as oryx/127.2.243.190@KRBTEST.COM (short username oryx)
I20260620 11:33:42.030329 9088 webserver.cc:460] Webserver started at http://127.2.243.190:43441/ using document root <none> and password file <none>
I20260620 11:33:42.030557 9088 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:42.030623 9088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:42.030782 9088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:42.031864 9088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "a5a8f84236d84cfb9ecad30874a8dc5a"
format_stamp: "Formatted at 2026-06-20 11:33:42 on dist-test-slave-ctbw"
server_key: "302b6ebbc3195933fd55aae877544a1e"
server_key_iv: "b213afb7da47632ea90cb44edd784e54"
server_key_version: "encryptionkey@0"
I20260620 11:33:42.032225 9088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "a5a8f84236d84cfb9ecad30874a8dc5a"
format_stamp: "Formatted at 2026-06-20 11:33:42 on dist-test-slave-ctbw"
server_key: "302b6ebbc3195933fd55aae877544a1e"
server_key_iv: "b213afb7da47632ea90cb44edd784e54"
server_key_version: "encryptionkey@0"
I20260620 11:33:42.033564 9088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:33:42.034354 9103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:42.034555 9088 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:42.034658 9088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "a5a8f84236d84cfb9ecad30874a8dc5a"
format_stamp: "Formatted at 2026-06-20 11:33:42 on dist-test-slave-ctbw"
server_key: "302b6ebbc3195933fd55aae877544a1e"
server_key_iv: "b213afb7da47632ea90cb44edd784e54"
server_key_version: "encryptionkey@0"
I20260620 11:33:42.034741 9088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:42.047950 9088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:42.050344 9088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:42.050510 9088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:42.054553 9088 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:41191
I20260620 11:33:42.054591 9155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:41191 every 8 connection(s)
I20260620 11:33:42.055002 9088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:33:42.055953 9156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:42.058233 9156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a: Bootstrap starting.
I20260620 11:33:42.058938 9156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:42.059103 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 9088
I20260620 11:33:42.059234 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:33:42.059300 9156 log.cc:824] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:42.059465 3022 external_mini_cluster.cc:1428] Setting key 1a014491e9337319d77f80c25d7e6034
I20260620 11:33:42.060181 9156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a: No bootstrap required, opened a new log
I20260620 11:33:42.062834 9156 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } }
I20260620 11:33:42.063050 9156 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:42.063083 9156 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5a8f84236d84cfb9ecad30874a8dc5a, State: Initialized, Role: FOLLOWER
I20260620 11:33:42.063213 9156 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } }
I20260620 11:33:42.063304 9156 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:33:42.063361 9156 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:33:42.063409 9156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:42.064003 9156 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } }
I20260620 11:33:42.064108 9156 leader_election.cc:302] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5a8f84236d84cfb9ecad30874a8dc5a; no voters:
I20260620 11:33:42.064258 9156 leader_election.cc:288] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:33:42.064285 9161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:42.064445 9161 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [term 1 LEADER]: Becoming Leader. State: Replica: a5a8f84236d84cfb9ecad30874a8dc5a, State: Running, Role: LEADER
I20260620 11:33:42.064498 9156 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:42.064545 9161 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } }
I20260620 11:33:42.064930 9162 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } }
I20260620 11:33:42.065009 9162 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:42.065219 9163 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5a8f84236d84cfb9ecad30874a8dc5a. Latest consensus state: current_term: 1 leader_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } }
I20260620 11:33:42.065332 9163 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:42.065501 9167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:33:42.066009 9167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 11:33:42 dist-test-slave-ctbw krb5kdc[9075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM
I20260620 11:33:42.067975 9167 catalog_manager.cc:1347] Generated new cluster ID: 0cbb43cecc8540a98dd3fee84da7a872
I20260620 11:33:42.068029 9167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:33:42.071614 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:42.060028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53258 (local address 127.2.243.190:41191)
0620 11:33:42.060212 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:42.060222 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:42.060267 (+ 45us) server_negotiation.cc:407] Connection header received
0620 11:33:42.060772 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:42.060781 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:42.060847 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:42.061004 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:42.061631 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.063086 (+ 1455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.063958 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.064632 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.065107 (+ 475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:42.067928 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:42.067931 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:42.067932 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:42.067945 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:42.069957 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.070382 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.070385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.070385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.070482 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.070694 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.070695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.070695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.071003 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:42.071149 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:42.071271 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 11:33:42.071399 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260620 11:33:42.081892 9167 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:33:42.082605 9167 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:33:42.087182 9167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a: Generated new TSK 0
I20260620 11:33:42.087363 9167 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.129.keytab.
Entry for principal oryx/127.2.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.129.keytab.
I20260620 11:33:42.114255 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:41191
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.129.keytab
--principal=oryx/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:34177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:42.194958 9182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:42.195204 9182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:42.195250 9182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:42.196445 9182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:42.196511 9182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:42.196555 9182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:33:42.198006 9182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:34177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.129.keytab
--principal=oryx/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:41191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:42.198275 9182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:42.198524 9182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:42.203924 9190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:42.203953 9187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:42.203975 9188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:42.204157 9182 server_base.cc:1034] running on GCE node
I20260620 11:33:42.204357 9182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:42.204592 9182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:42.205755 9182 hybrid_clock.cc:648] HybridClock initialized: now 1781955222205737 us; error 28 us; skew 500 ppm
Jun 20 11:33:42 dist-test-slave-ctbw krb5kdc[9075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955222, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:42.208165 9182 init.cc:377] Logged in from keytab as oryx/127.2.243.129@KRBTEST.COM (short username oryx)
I20260620 11:33:42.208609 9182 webserver.cc:460] Webserver started at http://127.2.243.129:43457/ using document root <none> and password file <none>
I20260620 11:33:42.208797 9182 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:42.208837 9182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:42.208964 9182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:42.210055 9182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "355a1e4b20b64b58864cc1ebd763ea12"
format_stamp: "Formatted at 2026-06-20 11:33:42 on dist-test-slave-ctbw"
server_key: "324e277e909e2632743ef4b9ca68b4a2"
server_key_iv: "7bb3e7cd55573eab43cadc134e0b7f13"
server_key_version: "encryptionkey@0"
I20260620 11:33:42.210424 9182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "355a1e4b20b64b58864cc1ebd763ea12"
format_stamp: "Formatted at 2026-06-20 11:33:42 on dist-test-slave-ctbw"
server_key: "324e277e909e2632743ef4b9ca68b4a2"
server_key_iv: "7bb3e7cd55573eab43cadc134e0b7f13"
server_key_version: "encryptionkey@0"
I20260620 11:33:42.211812 9182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:33:42.212517 9197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:42.212700 9182 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:42.212780 9182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "355a1e4b20b64b58864cc1ebd763ea12"
format_stamp: "Formatted at 2026-06-20 11:33:42 on dist-test-slave-ctbw"
server_key: "324e277e909e2632743ef4b9ca68b4a2"
server_key_iv: "7bb3e7cd55573eab43cadc134e0b7f13"
server_key_version: "encryptionkey@0"
I20260620 11:33:42.212831 9182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:42.221122 9182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:42.223054 9182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:42.223160 9182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:42.223331 9182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:42.223556 9182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:42.223585 9182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:42.223605 9182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:42.223618 9182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:42.229005 9182 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:42977
I20260620 11:33:42.229092 9310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:42977 every 8 connection(s)
I20260620 11:33:42.229349 9182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
Jun 20 11:33:42 dist-test-slave-ctbw krb5kdc[9075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955222, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM
I20260620 11:33:42.239300 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:42.230058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:43275 (local address 127.2.243.190:41191)
0620 11:33:42.230144 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:42.230147 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:42.230342 (+ 195us) server_negotiation.cc:407] Connection header received
0620 11:33:42.230981 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:42.230983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:42.230997 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:42.231064 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:42.232063 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.232593 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.233636 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.233870 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.233959 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:42.236338 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:42.236340 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:42.236340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:42.236348 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:42.237903 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.238407 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.238408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.238409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.238475 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.238700 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.238701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.238701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.238813 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:42.238908 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:42.239147 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 11:33:42.239231 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 11:33:42.239393 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 9182
I20260620 11:33:42.239482 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:33:42.239583 9313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:42.230027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.2.243.129:43275)
0620 11:33:42.230198 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:42.230243 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:42.230794 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:42.231135 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:42.231139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:42.231752 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:42.231943 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:42.231948 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.232703 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.232705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:42.233522 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:42.233524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.233970 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.233971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:42.234101 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:42.234679 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:42.234687 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:42.236220 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:42.238000 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:42.238006 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:42.238008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:42.238274 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:42.238536 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:42.238537 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:42.238537 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:42.238641 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:42.238904 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:42.238914 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:42.239033 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 11:33:42.239121 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 11:33:42.239320 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260620 11:33:42.239713 3022 external_mini_cluster.cc:1428] Setting key 18640d54bab40c185e14de93e0429e88
I20260620 11:33:42.240264 9311 heartbeater.cc:347] Connected to a master server at 127.2.243.190:41191
I20260620 11:33:42.240368 9311 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:42.240547 9311 heartbeater.cc:511] Master 127.2.243.190:41191 requested a full tablet report, sending...
I20260620 11:33:42.241106 9119 ts_manager.cc:194] Registered new tserver with Master: 355a1e4b20b64b58864cc1ebd763ea12 (127.2.243.129:42977)
I20260620 11:33:42.241719 9119 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:43275
WARNING: no policy specified for oryx/127.2.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.130.keytab.
Entry for principal oryx/127.2.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.130.keytab.
I20260620 11:33:42.265596 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:41191
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.130.keytab
--principal=oryx/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:34177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:42.346599 9316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:42.346817 9316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:42.346840 9316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:42.348048 9316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:42.348095 9316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:42.348122 9316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:33:42.349642 9316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:34177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.130.keytab
--principal=oryx/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:41191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:42.349891 9316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:42.350135 9316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:42.355278 9322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:42.355269 9324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:42.355269 9321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:42.355654 9316 server_base.cc:1034] running on GCE node
I20260620 11:33:42.355909 9316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:42.356158 9316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:42.357317 9316 hybrid_clock.cc:648] HybridClock initialized: now 1781955222357288 us; error 38 us; skew 500 ppm
Jun 20 11:33:42 dist-test-slave-ctbw krb5kdc[9075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955222, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:42.360042 9316 init.cc:377] Logged in from keytab as oryx/127.2.243.130@KRBTEST.COM (short username oryx)
I20260620 11:33:42.360507 9316 webserver.cc:460] Webserver started at http://127.2.243.130:40939/ using document root <none> and password file <none>
I20260620 11:33:42.360775 9316 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:42.360849 9316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:42.361004 9316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:42.362100 9316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "a97ab72ec734489c8cff99acbc81b178"
format_stamp: "Formatted at 2026-06-20 11:33:42 on dist-test-slave-ctbw"
server_key: "de2fcc80fb6723fbcd2d176296df029f"
server_key_iv: "8c6f64736ad18282e6a90d4c1a5ff2e1"
server_key_version: "encryptionkey@0"
I20260620 11:33:42.362489 9316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "a97ab72ec734489c8cff99acbc81b178"
format_stamp: "Formatted at 2026-06-20 11:33:42 on dist-test-slave-ctbw"
server_key: "de2fcc80fb6723fbcd2d176296df029f"
server_key_iv: "8c6f64736ad18282e6a90d4c1a5ff2e1"
server_key_version: "encryptionkey@0"
I20260620 11:33:42.364162 9316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:33:42.364960 9331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:42.365190 9316 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:42.365283 9316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "a97ab72ec734489c8cff99acbc81b178"
format_stamp: "Formatted at 2026-06-20 11:33:42 on dist-test-slave-ctbw"
server_key: "de2fcc80fb6723fbcd2d176296df029f"
server_key_iv: "8c6f64736ad18282e6a90d4c1a5ff2e1"
server_key_version: "encryptionkey@0"
I20260620 11:33:42.365341 9316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:42.378836 9316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:42.380980 9316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:42.381093 9316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:42.381268 9316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:42.381596 9316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:42.381630 9316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:42.381680 9316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:42.381707 9316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:42.387394 9316 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:41803
I20260620 11:33:42.387475 9444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:41803 every 8 connection(s)
I20260620 11:33:42.387784 9316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:33:42.390468 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 9316
I20260620 11:33:42.390586 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:33:42.390774 3022 external_mini_cluster.cc:1428] Setting key f405e6aad14d09d1e7073d48bcf528b5
Jun 20 11:33:42 dist-test-slave-ctbw krb5kdc[9075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955222, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM
I20260620 11:33:42.398418 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:42.388563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:52423 (local address 127.2.243.190:41191)
0620 11:33:42.388639 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:42.388641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:42.388847 (+ 206us) server_negotiation.cc:407] Connection header received
0620 11:33:42.389474 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:42.389476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:42.389489 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:42.389551 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:42.390574 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.391284 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.392352 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.392568 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.392662 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:42.395544 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:42.395546 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:42.395546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:42.395556 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:42.397037 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.397453 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.397454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.397455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.397516 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.397774 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.397775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.397776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.397893 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:42.397968 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:42.398241 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 11:33:42.398319 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 11:33:42.398655 9447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:42.388548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.2.243.130:52423)
0620 11:33:42.388703 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:42.388751 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:42.389348 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:42.389624 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:42.389628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:42.390248 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:42.390454 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:42.390459 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.391398 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.391399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:42.392240 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:42.392243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.392680 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.392681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:42.392899 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:42.393539 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:42.393551 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:42.395427 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:42.397140 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:42.397147 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:42.397149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:42.397376 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:42.397582 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:42.397583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:42.397583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:42.397704 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:42.397959 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:42.397965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:42.398109 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 11:33:42.398219 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 11:33:42.398388 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260620 11:33:42.399111 9445 heartbeater.cc:347] Connected to a master server at 127.2.243.190:41191
I20260620 11:33:42.399223 9445 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.2.243.131@KRBTEST.COM; defaulting to no policy
I20260620 11:33:42.399420 9445 heartbeater.cc:511] Master 127.2.243.190:41191 requested a full tablet report, sending...
I20260620 11:33:42.399842 9119 ts_manager.cc:194] Registered new tserver with Master: a97ab72ec734489c8cff99acbc81b178 (127.2.243.130:41803)
I20260620 11:33:42.400235 9119 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:52423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.131.keytab.
Entry for principal oryx/127.2.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.131.keytab.
I20260620 11:33:42.418074 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:41191
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.131.keytab
--principal=oryx/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:34177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:42.499505 9450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:42.499709 9450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:42.499732 9450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:42.500780 9450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:42.500813 9450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:42.500837 9450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:33:42.502197 9450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:34177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.131.keytab
--principal=oryx/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:41191
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:42.502382 9450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:42.502619 9450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:42.507532 9456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:42.507511 9458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:42.507511 9455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:42.507718 9450 server_base.cc:1034] running on GCE node
I20260620 11:33:42.507992 9450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:42.508234 9450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:42.509383 9450 hybrid_clock.cc:648] HybridClock initialized: now 1781955222509370 us; error 30 us; skew 500 ppm
Jun 20 11:33:42 dist-test-slave-ctbw krb5kdc[9075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955222, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:42.511914 9450 init.cc:377] Logged in from keytab as oryx/127.2.243.131@KRBTEST.COM (short username oryx)
I20260620 11:33:42.512293 9450 webserver.cc:460] Webserver started at http://127.2.243.131:37007/ using document root <none> and password file <none>
I20260620 11:33:42.512485 9450 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:42.512526 9450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:42.512646 9450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:42.513649 9450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "c3ec40c3f3a44f958349012a3bda33e1"
format_stamp: "Formatted at 2026-06-20 11:33:42 on dist-test-slave-ctbw"
server_key: "57c6ba798ac4047d60ec18c186a3fbaa"
server_key_iv: "67462b5dfaf1e23f5ccae987ed47e64b"
server_key_version: "encryptionkey@0"
I20260620 11:33:42.513949 9450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "c3ec40c3f3a44f958349012a3bda33e1"
format_stamp: "Formatted at 2026-06-20 11:33:42 on dist-test-slave-ctbw"
server_key: "57c6ba798ac4047d60ec18c186a3fbaa"
server_key_iv: "67462b5dfaf1e23f5ccae987ed47e64b"
server_key_version: "encryptionkey@0"
I20260620 11:33:42.515291 9450 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:33:42.516026 9465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:42.516184 9450 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:33:42.516287 9450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "c3ec40c3f3a44f958349012a3bda33e1"
format_stamp: "Formatted at 2026-06-20 11:33:42 on dist-test-slave-ctbw"
server_key: "57c6ba798ac4047d60ec18c186a3fbaa"
server_key_iv: "67462b5dfaf1e23f5ccae987ed47e64b"
server_key_version: "encryptionkey@0"
I20260620 11:33:42.516371 9450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:42.542114 9450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:42.544662 9450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:42.544833 9450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:42.545086 9450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:42.545403 9450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:42.545457 9450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:42.545499 9450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:42.545527 9450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:42.551182 9450 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:46759
I20260620 11:33:42.551231 9578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:46759 every 8 connection(s)
I20260620 11:33:42.551657 9450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:33:42.553268 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 9450
I20260620 11:33:42.553386 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:33:42.553570 3022 external_mini_cluster.cc:1428] Setting key 7dec9053a0ee2e574ac632ebac89d180
Jun 20 11:33:42 dist-test-slave-ctbw krb5kdc[9075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955222, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM
I20260620 11:33:42.561107 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:42.552404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:53117 (local address 127.2.243.190:41191)
0620 11:33:42.552449 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:42.552451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:42.552691 (+ 240us) server_negotiation.cc:407] Connection header received
0620 11:33:42.553225 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:42.553226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:42.553239 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:42.553289 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:42.554333 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.554822 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.556039 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.556277 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.556369 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:42.558463 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:42.558464 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:42.558465 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:42.558473 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:42.559736 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.560171 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.560172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.560172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.560237 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.560477 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.560478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.560479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.560593 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:42.560664 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:42.560935 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 11:33:42.561013 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:33:42.561311 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:42.552395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.2.243.131:53117)
0620 11:33:42.552565 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:42.552602 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:42.553112 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:42.553389 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:42.553394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:42.554031 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:42.554224 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:42.554229 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.554983 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.554984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:42.555810 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:42.555811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.556379 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.556381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:42.556513 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:42.556989 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:42.556999 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:42.558370 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:42.559857 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:42.559864 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:42.559867 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:42.560091 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:42.560305 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:42.560306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:42.560306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:42.560411 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:42.560664 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:42.560671 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:42.560810 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 11:33:42.560915 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 11:33:42.561074 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260620 11:33:42.561782 9579 heartbeater.cc:347] Connected to a master server at 127.2.243.190:41191
I20260620 11:33:42.561879 9579 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:42.562063 9579 heartbeater.cc:511] Master 127.2.243.190:41191 requested a full tablet report, sending...
I20260620 11:33:42.562436 9119 ts_manager.cc:194] Registered new tserver with Master: c3ec40c3f3a44f958349012a3bda33e1 (127.2.243.131:46759)
I20260620 11:33:42.562804 9119 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:53117
I20260620 11:33:42.565187 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.243.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.243.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.243.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.189.keytab.
Entry for principal oryx/127.2.243.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.189.keytab.
I20260620 11:33:42.589247 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.189:38923
--webserver_interface=127.2.243.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.189.keytab
--principal=oryx/127.2.243.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:34177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:41191,127.2.243.189:38923
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:42.670028 9584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:42.670279 9584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:42.670318 9584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:42.671506 9584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:33:42.671574 9584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:42.671597 9584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:42.671617 9584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:33:42.671650 9584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:33:42.673143 9584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:34177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:41191,127.2.243.189:38923
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.189.keytab
--principal=oryx/127.2.243.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.189:38923
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.243.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
W20260620 11:33:42.673291 9584 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.2.243.190:41191,127.2.243.189:38923'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 11:33:42.673445 9584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:42.673662 9584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:42.678494 9589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:42.678594 9584 server_base.cc:1034] running on GCE node
W20260620 11:33:42.678601 9590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:42.678526 9592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:42.678943 9584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:42.679250 9584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:42.680454 9584 hybrid_clock.cc:648] HybridClock initialized: now 1781955222680440 us; error 27 us; skew 500 ppm
Jun 20 11:33:42 dist-test-slave-ctbw krb5kdc[9075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955222, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:42.682807 9584 init.cc:377] Logged in from keytab as oryx/127.2.243.189@KRBTEST.COM (short username oryx)
I20260620 11:33:42.683318 9584 webserver.cc:460] Webserver started at http://127.2.243.189:37395/ using document root <none> and password file <none>
I20260620 11:33:42.683557 9584 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:42.683625 9584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:42.683774 9584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:42.685122 9584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/data/instance:
uuid: "3aa17d267b434b8781609a0171af2bca"
format_stamp: "Formatted at 2026-06-20 11:33:42 on dist-test-slave-ctbw"
server_key: "aab8d465a54251e12852696fac8232c0"
server_key_iv: "a0f3b8ed1a79fdf1b033bc4fece302a6"
server_key_version: "encryptionkey@0"
I20260620 11:33:42.685509 9584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/wal/instance:
uuid: "3aa17d267b434b8781609a0171af2bca"
format_stamp: "Formatted at 2026-06-20 11:33:42 on dist-test-slave-ctbw"
server_key: "aab8d465a54251e12852696fac8232c0"
server_key_iv: "a0f3b8ed1a79fdf1b033bc4fece302a6"
server_key_version: "encryptionkey@0"
I20260620 11:33:42.686827 9584 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:33:42.687688 9599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:42.687881 9584 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:42.687980 9584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/wal
uuid: "3aa17d267b434b8781609a0171af2bca"
format_stamp: "Formatted at 2026-06-20 11:33:42 on dist-test-slave-ctbw"
server_key: "aab8d465a54251e12852696fac8232c0"
server_key_iv: "a0f3b8ed1a79fdf1b033bc4fece302a6"
server_key_version: "encryptionkey@0"
I20260620 11:33:42.688062 9584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:42.709973 9584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:42.711990 9584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:42.712152 9584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:42.716311 9584 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.189:38923
I20260620 11:33:42.716343 9651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.189:38923 every 8 connection(s)
I20260620 11:33:42.716714 9584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/data/info.pb
I20260620 11:33:42.717689 9652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:42.719394 9652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:33:42.724195 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 9584
Jun 20 11:33:42 dist-test-slave-ctbw krb5kdc[9075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955222, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.189@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM
I20260620 11:33:42.729120 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:42.720250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44024 (local address 127.2.243.190:41191)
0620 11:33:42.720368 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:42.720370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:42.720546 (+ 176us) server_negotiation.cc:407] Connection header received
0620 11:33:42.721090 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:42.721091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:42.721103 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:42.721175 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:42.721816 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.722292 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.723481 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.723664 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.723741 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:42.726103 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:42.726105 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:42.726105 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:42.726113 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:42.727524 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.728056 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.728057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.728058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.728125 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.728377 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.728378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.728378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.728602 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:42.728685 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:42.728916 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 11:33:42.728994 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 11:33:42.729161 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:42.720238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.0.0.1:44024)
0620 11:33:42.720424 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:42.720463 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:42.720926 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:42.721254 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:42.721258 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:42.721519 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:42.721717 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:42.721721 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.722374 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.722375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:42.723400 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:42.723402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.723754 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.723755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:42.723870 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:42.724426 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:42.724443 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:42.726003 (+ 1560us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:42.727624 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:42.727631 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:42.727633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:42.727955 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:42.728204 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:42.728205 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:42.728205 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:42.728307 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:42.728676 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:42.728680 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:42.728787 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 11:33:42.728868 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 11:33:42.729000 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
Jun 20 11:33:42 dist-test-slave-ctbw krb5kdc[9075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.243.189@KRBTEST.COM
I20260620 11:33:42.729806 9652 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:33:42.733246 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:42.724728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53598 (local address 127.2.243.189:38923)
0620 11:33:42.724875 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:42.724879 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:42.724955 (+ 76us) server_negotiation.cc:407] Connection header received
0620 11:33:42.725015 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:42.725016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:42.725036 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:42.725100 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:42.725713 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.726375 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.727171 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.727470 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.727558 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:42.730222 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:42.730229 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:42.730229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:42.730236 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:42.731846 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.732232 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.732235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.732235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.732315 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.732541 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.732542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.732542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.732688 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:42.732800 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:42.732918 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 11:33:42.732976 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
Jun 20 11:33:42 dist-test-slave-ctbw krb5kdc[9075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955222, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.189@KRBTEST.COM for oryx/127.2.243.189@KRBTEST.COM
I20260620 11:33:42.736287 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:42.730099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:38923 (local address 127.0.0.1:53604)
0620 11:33:42.730130 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:42.730139 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:42.730177 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:42.730375 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:42.730376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:42.730516 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:42.730552 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:42.730553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.731283 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.731284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:42.731742 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:42.731743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.732125 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.732125 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:42.732185 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:42.732559 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:42.732564 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:42.733957 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:42.735321 (+ 1364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:42.735322 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:42.735322 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:42.735479 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:42.735678 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:42.735679 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:42.735680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:42.735730 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:42.735970 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:42.735971 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:42.736005 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 11:33:42.736072 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 11:33:42.736163 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 11:33:42.736359 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:42.730132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53604 (local address 127.2.243.189:38923)
0620 11:33:42.730263 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:42.730265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:42.730271 (+ 6us) server_negotiation.cc:407] Connection header received
0620 11:33:42.730294 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:42.730295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:42.730302 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:42.730373 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:42.730630 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.731191 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.731837 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.732044 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.732126 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:42.734045 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:42.734046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:42.734046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:42.734052 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:42.735239 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.735559 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.735560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.735561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.735615 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.735804 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.735805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.735805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.735906 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:42.735978 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:42.736099 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 11:33:42.736213 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260620 11:33:42.737653 9652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca: Bootstrap starting.
I20260620 11:33:42.738267 9652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:42.738543 9652 log.cc:824] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:42.739292 9652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca: No bootstrap required, opened a new log
I20260620 11:33:42.740761 9652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } }
I20260620 11:33:42.740895 9652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:42.740918 9652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aa17d267b434b8781609a0171af2bca, State: Initialized, Role: FOLLOWER
I20260620 11:33:42.741077 9652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } }
I20260620 11:33:42.741333 9659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } } }
I20260620 11:33:42.741381 9652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:42.741442 9659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [sys.catalog]: This master's current role is: FOLLOWER
W20260620 11:33:42.743113 9670 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:33:42.743170 9670 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:33:42.744829 9584 master_runner.cc:428] Detected that this master 3aa17d267b434b8781609a0171af2bca is joining an existing cluster
I20260620 11:33:42.744882 9584 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.243.189:38923
I20260620 11:33:42.753127 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:42.746182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53616 (local address 127.2.243.189:38923)
0620 11:33:42.746219 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:42.746221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:42.746322 (+ 101us) server_negotiation.cc:407] Connection header received
0620 11:33:42.746426 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:42.746426 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:42.746438 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:42.746492 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:42.746818 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.747333 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.748391 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.748709 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.748787 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:42.750284 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:42.750286 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:42.750286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:42.750293 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:42.751976 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.752343 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.752344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.752344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.752393 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.752601 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.752602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.752602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.752714 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:42.752787 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:42.752933 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 11:33:42.753019 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:33:42.753403 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:42.746127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:38923 (local address 127.0.0.1:53616)
0620 11:33:42.746263 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:42.746270 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:42.746331 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:42.746501 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:42.746502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:42.746685 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:42.746735 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:42.746736 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.747414 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.747416 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:42.748248 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:42.748250 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.748808 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.748809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:42.748874 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:42.749545 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:42.749549 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:42.750184 (+ 635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:42.752076 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:42.752077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:42.752078 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:42.752258 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:42.752482 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:42.752482 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:42.752483 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:42.752539 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:42.752778 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:42.752779 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:42.752817 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 11:33:42.752914 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 11:33:42.753074 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":63,"thread_start_us":75,"threads_started":1}
I20260620 11:33:42.753645 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:42.745891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.0.0.1:44040)
0620 11:33:42.746054 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:42.746066 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:42.746126 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:42.746352 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:42.746353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:42.746619 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:42.746704 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:42.746707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.747695 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.747696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:42.748458 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:42.748460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.748858 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.748859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:42.748910 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:42.749627 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:42.749631 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:42.750184 (+ 553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:42.752508 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:42.752509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:42.752509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:42.752716 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:42.752926 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:42.752927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:42.752927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:42.752982 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:42.753257 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:42.753258 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:42.753360 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 11:33:42.753434 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 11:33:42.753532 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260620 11:33:42.753890 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:42.745950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44040 (local address 127.2.243.190:41191)
0620 11:33:42.745998 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:42.746000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:42.746212 (+ 212us) server_negotiation.cc:407] Connection header received
0620 11:33:42.746238 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:42.746239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:42.746250 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:42.746329 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:42.746795 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.747609 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.748538 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.748730 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.748808 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:42.750558 (+ 1750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:42.750560 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:42.750560 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:42.750567 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:42.752306 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.752800 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.752801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.752802 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.752859 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.753050 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.753051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.753051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.753172 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:42.753232 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:42.753579 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 11:33:42.753661 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Jun 20 11:33:42 dist-test-slave-ctbw krb5kdc[9075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955222, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.190@KRBTEST.COM for oryx/127.2.243.189@KRBTEST.COM
I20260620 11:33:42.763962 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:42.756285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:38923 (local address 127.0.0.1:53626)
0620 11:33:42.756484 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:42.756527 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:42.756596 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:42.756744 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:42.756746 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:42.756990 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:42.757056 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:42.757057 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.757959 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.757960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:42.758565 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:42.758567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.759049 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.759050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:42.759106 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:42.759559 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:42.759563 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:42.761153 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:42.762804 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:42.762811 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:42.762812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:42.763076 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:42.763271 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:42.763272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:42.763272 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:42.763331 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:42.763601 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:42.763607 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:42.763650 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 11:33:42.763714 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 11:33:42.763805 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":137,"threads_started":1}
I20260620 11:33:42.763967 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:42.756390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53626 (local address 127.2.243.189:38923)
0620 11:33:42.756445 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:42.756447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:42.756593 (+ 146us) server_negotiation.cc:407] Connection header received
0620 11:33:42.756652 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:42.756653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:42.756663 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:42.756741 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:42.757124 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.757878 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.758645 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.758980 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.759051 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:42.761237 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:42.761238 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:42.761238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:42.761246 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:42.762721 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.763145 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.763146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.763147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.763204 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.763399 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.763400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.763400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.763535 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:42.763605 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:42.763757 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 11:33:42.763832 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 11:33:42.764845 9119 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.243.189:38923: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3aa17d267b434b8781609a0171af2bca"
member_type: NON_VOTER
last_known_addr {
host: "127.2.243.189"
port: 38923
}
attrs {
promote: true
}
}
dest_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a"
cas_config_opid_index: -1
I20260620 11:33:42.765192 9119 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: NON_VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: true } }
I20260620 11:33:42.765942 9680 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3aa17d267b434b8781609a0171af2bca (127.2.243.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: NON_VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: true } } }
I20260620 11:33:42.766271 9626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:42.767030 9680 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.243.189:38923
I20260620 11:33:42.767254 9584 master.cc:537] Master@127.2.243.189:38923 shutting down...
I20260620 11:33:42.767309 9626 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 1 FOLLOWER]: Refusing update from remote peer a5a8f84236d84cfb9ecad30874a8dc5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 11:33:42.768690 9680 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: NON_VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:42.768718 9679 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: NON_VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: true } } }
I20260620 11:33:42.768806 9679 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [sys.catalog]: This master's current role is: LEADER
W20260620 11:33:42.769871 9107 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 11:33:42.770278 9107 consensus_peers.cc:487] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a -> Peer 3aa17d267b434b8781609a0171af2bca (127.2.243.189:38923): Couldn't send request to peer 3aa17d267b434b8781609a0171af2bca. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 11:33:42.770999 9584 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:42.771077 9584 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:42.771145 9584 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca: stopping tablet replica
I20260620 11:33:42.784163 9584 master.cc:552] Master@127.2.243.189:38923 shutdown complete.
I20260620 11:33:42.784229 9584 master_runner.cc:305] Clearing existing system tablet
I20260620 11:33:42.784368 9584 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:42.784425 9584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:42.785064 9584 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.001s
I20260620 11:33:42.785599 9684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:42.785763 9584 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:42.785825 9584 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:42.785848 9584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/wal
uuid: "3aa17d267b434b8781609a0171af2bca"
format_stamp: "Formatted at 2026-06-20 11:33:42 on dist-test-slave-ctbw"
server_key: "aab8d465a54251e12852696fac8232c0"
server_key_iv: "a0f3b8ed1a79fdf1b033bc4fece302a6"
server_key_version: "encryptionkey@0"
I20260620 11:33:42.786484 9584 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:42.788039 9584 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 11:33:42.788115 9584 log.cc:1196] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 11:33:42.788378 9584 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca: Deleting consensus metadata
I20260620 11:33:42.788529 9584 master_runner.cc:315] Copying system tablet from 127.2.243.190:41191
I20260620 11:33:42.789265 9584 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca: tablet copy: Beginning tablet copy session from remote peer at address 127.2.243.190:41191
I20260620 11:33:42.796285 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:42.789684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44042 (local address 127.2.243.190:41191)
0620 11:33:42.789742 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:42.789744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:42.789816 (+ 72us) server_negotiation.cc:407] Connection header received
0620 11:33:42.789888 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:42.789889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:42.789898 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:42.789954 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:42.790341 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.790917 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.792021 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.792213 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.792299 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:42.793358 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:42.793359 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:42.793360 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:42.793366 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:42.795075 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.795458 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.795459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.795459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.795514 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.795713 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.795713 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.795714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.795837 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:42.795895 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:42.796089 (+ 194us) server_negotiation.cc:299] Negotiation successful
0620 11:33:42.796173 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 11:33:42.796605 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:42.789590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.0.0.1:44042)
0620 11:33:42.789734 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:42.789746 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:42.789824 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:42.789991 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:42.789992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:42.790187 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:42.790254 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:42.790256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.791011 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.791013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:42.791914 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:42.791917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.792309 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.792310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:42.792380 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:42.792858 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:42.792864 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:42.793253 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:42.795151 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:42.795152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:42.795153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:42.795386 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:42.795579 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:42.795579 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:42.795580 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:42.795650 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:42.795903 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:42.795904 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:42.795950 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 11:33:42.796385 (+ 435us) client_negotiation.cc:240] Negotiation successful
0620 11:33:42.796501 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260620 11:33:42.797019 9140 tablet_copy_service.cc:140] P a5a8f84236d84cfb9ecad30874a8dc5a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3aa17d267b434b8781609a0171af2bca ({username='oryx', principal='oryx/127.2.243.189@KRBTEST.COM'} at 127.0.0.1:44042)
I20260620 11:33:42.797122 9140 tablet_copy_service.cc:161] P a5a8f84236d84cfb9ecad30874a8dc5a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3aa17d267b434b8781609a0171af2bca at {username='oryx', principal='oryx/127.2.243.189@KRBTEST.COM'} at 127.0.0.1:44042: session id = 3aa17d267b434b8781609a0171af2bca-00000000000000000000000000000000
I20260620 11:33:42.797952 9140 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 11:33:42.798573 9584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 11:33:42.799715 9584 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca: tablet copy: Starting download of 0 data blocks...
I20260620 11:33:42.799927 9584 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca: tablet copy: Starting download of 1 WAL segments...
I20260620 11:33:42.800936 9584 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 11:33:42.801810 9140 tablet_copy_service.cc:342] P a5a8f84236d84cfb9ecad30874a8dc5a: Request end of tablet copy session 3aa17d267b434b8781609a0171af2bca-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.243.189@KRBTEST.COM'} at 127.0.0.1:44042
I20260620 11:33:42.801867 9140 tablet_copy_service.cc:433] P a5a8f84236d84cfb9ecad30874a8dc5a: ending tablet copy session 3aa17d267b434b8781609a0171af2bca-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3aa17d267b434b8781609a0171af2bca
I20260620 11:33:42.804414 9584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:42.805639 9692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:42.805647 9691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:42.807701 9694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:42.807838 9584 server_base.cc:1034] running on GCE node
I20260620 11:33:42.807951 9584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:42.808094 9584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:42.809221 9584 hybrid_clock.cc:648] HybridClock initialized: now 1781955222809214 us; error 25 us; skew 500 ppm
Jun 20 11:33:42 dist-test-slave-ctbw krb5kdc[9075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955222, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:42.810678 9584 init.cc:377] Logged in from keytab as oryx/127.2.243.189@KRBTEST.COM (short username oryx)
I20260620 11:33:42.810974 9584 webserver.cc:460] Webserver started at http://127.2.243.189:43701/ using document root <none> and password file <none>
I20260620 11:33:42.811103 9584 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:42.811151 9584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:42.811869 9584 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:42.812438 9701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:42.812597 9584 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:42.812659 9584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/wal
uuid: "3aa17d267b434b8781609a0171af2bca"
format_stamp: "Formatted at 2026-06-20 11:33:42 on dist-test-slave-ctbw"
server_key: "aab8d465a54251e12852696fac8232c0"
server_key_iv: "a0f3b8ed1a79fdf1b033bc4fece302a6"
server_key_version: "encryptionkey@0"
I20260620 11:33:42.812732 9584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:42.833902 9584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:42.835726 9584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:42.839082 9584 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.189:38923
I20260620 11:33:42.839103 9753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.189:38923 every 8 connection(s)
I20260620 11:33:42.839432 9584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-1/data/info.pb
I20260620 11:33:42.840318 9754 sys_catalog.cc:263] Verifying existing consensus state
I20260620 11:33:42.840863 9754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca: Bootstrap starting.
I20260620 11:33:42.843750 9754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:33:42.843979 9754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca: Bootstrap complete.
I20260620 11:33:42.844173 9754 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: NON_VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: true } }
I20260620 11:33:42.844264 9754 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3aa17d267b434b8781609a0171af2bca, State: Initialized, Role: LEARNER
I20260620 11:33:42.844316 9754 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: NON_VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: true } }
I20260620 11:33:42.844514 9754 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:42.844506 9757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: NON_VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: true } } }
I20260620 11:33:42.844623 9757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [sys.catalog]: This master's current role is: LEARNER
I20260620 11:33:42.846045 9768 catalog_manager.cc:1259] Loaded cluster ID: 0cbb43cecc8540a98dd3fee84da7a872
I20260620 11:33:42.846155 9768 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca: loading cluster ID for follower catalog manager: success
I20260620 11:33:42.847182 9768 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca: acquiring CA information for follower catalog manager: success
I20260620 11:33:42.847406 9768 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 11:33:42.891983 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:42.885372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53656 (local address 127.2.243.189:38923)
0620 11:33:42.885510 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:42.885512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:42.885521 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:33:42.885548 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:42.885549 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:42.885564 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:42.885639 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:42.886200 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.886848 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.887614 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:42.887808 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:42.887889 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:42.889196 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:42.889198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:42.889198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:42.889206 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:42.890747 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.891121 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.891122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.891122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.891183 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:42.891435 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:42.891436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:42.891436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:42.891592 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:42.891693 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:42.891800 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 11:33:42.891881 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
WARNING: no policy specified for oryx/127.2.243.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.243.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.243.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.188.keytab.
Entry for principal oryx/127.2.243.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.188.keytab.
I20260620 11:33:42.916193 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.188:38025
--webserver_interface=127.2.243.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.188.keytab
--principal=oryx/127.2.243.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:34177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:41191,127.2.243.189:38923,127.2.243.188:38025
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:42.997344 9772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:42.997581 9772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:42.997620 9772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:42.998783 9772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:33:42.998853 9772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:42.998875 9772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:42.998926 9772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:33:42.998955 9772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:33:43.000448 9772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:34177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:41191,127.2.243.189:38923,127.2.243.188:38025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.188.keytab
--principal=oryx/127.2.243.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.188:38025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.243.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:43.000718 9772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:43.000947 9772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:43.005882 9778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:43.005867 9780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:43.005863 9777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:43.006187 9772 server_base.cc:1034] running on GCE node
I20260620 11:33:43.006505 9772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:43.006794 9772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:43.007956 9772 hybrid_clock.cc:648] HybridClock initialized: now 1781955223007947 us; error 33 us; skew 500 ppm
Jun 20 11:33:43 dist-test-slave-ctbw krb5kdc[9075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955223, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:43.010416 9772 init.cc:377] Logged in from keytab as oryx/127.2.243.188@KRBTEST.COM (short username oryx)
I20260620 11:33:43.010948 9772 webserver.cc:460] Webserver started at http://127.2.243.188:40471/ using document root <none> and password file <none>
I20260620 11:33:43.011191 9772 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:43.011236 9772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:43.011324 9772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:43.012351 9772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/data/instance:
uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23"
format_stamp: "Formatted at 2026-06-20 11:33:43 on dist-test-slave-ctbw"
server_key: "013b2efca99e6ef716865dc6dfa6c057"
server_key_iv: "b1a374bef8f89c9871299d2b21414605"
server_key_version: "encryptionkey@0"
I20260620 11:33:43.012707 9772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/wal/instance:
uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23"
format_stamp: "Formatted at 2026-06-20 11:33:43 on dist-test-slave-ctbw"
server_key: "013b2efca99e6ef716865dc6dfa6c057"
server_key_iv: "b1a374bef8f89c9871299d2b21414605"
server_key_version: "encryptionkey@0"
I20260620 11:33:43.014014 9772 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:33:43.014823 9787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:43.015039 9772 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:43.015132 9772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/wal
uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23"
format_stamp: "Formatted at 2026-06-20 11:33:43 on dist-test-slave-ctbw"
server_key: "013b2efca99e6ef716865dc6dfa6c057"
server_key_iv: "b1a374bef8f89c9871299d2b21414605"
server_key_version: "encryptionkey@0"
I20260620 11:33:43.015214 9772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:43.029352 9772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:43.031816 9772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:43.031972 9772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:43.036366 9772 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.188:38025
I20260620 11:33:43.036427 9839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.188:38025 every 8 connection(s)
I20260620 11:33:43.036775 9772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/data/info.pb
I20260620 11:33:43.037963 9840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:43.039880 9840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:33:43.041227 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 9772
Jun 20 11:33:43 dist-test-slave-ctbw krb5kdc[9075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955223, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.188@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM
Jun 20 11:33:43 dist-test-slave-ctbw krb5kdc[9075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.243.188@KRBTEST.COM
I20260620 11:33:43.050500 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.040748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44056 (local address 127.2.243.190:41191)
0620 11:33:43.040815 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.040817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.040859 (+ 42us) server_negotiation.cc:407] Connection header received
0620 11:33:43.041709 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.041711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.041726 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.041822 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.042691 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.043425 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.044641 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.044832 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.044919 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.047478 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.047479 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.047479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.047487 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.049092 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.049541 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.049542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.049542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.049608 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.049864 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.049865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.049865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.049987 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.050061 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.050324 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.050405 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 11:33:43.050683 9842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.040557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.0.0.1:44056)
0620 11:33:43.040747 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.040789 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.041563 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.041917 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.041922 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.042285 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:43.042583 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.042589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.043560 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.043561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.044533 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.044536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.044979 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.044980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.045101 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.045674 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:43.045683 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:43.047395 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:43.049204 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.049210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.049212 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.049462 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.049675 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.049676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.049676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.049786 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.050053 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:43.050059 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:43.050193 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.050303 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.050451 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":69,"threads_started":1}
I20260620 11:33:43.050808 9843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.041813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51990 (local address 127.2.243.188:38025)
0620 11:33:43.041999 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.042004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.042029 (+ 25us) server_negotiation.cc:407] Connection header received
0620 11:33:43.042050 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.042052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.042073 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.042155 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.042847 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.043739 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.044443 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.044770 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.044880 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.047977 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.047984 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.047984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.047992 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.049603 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.049975 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.049978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.049978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.050036 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.050226 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.050227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.050227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.050371 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.050533 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.050580 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.050669 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":124,"threads_started":1}
I20260620 11:33:43.051265 9840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 11:33:43 dist-test-slave-ctbw krb5kdc[9075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955223, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.188@KRBTEST.COM for oryx/127.2.243.189@KRBTEST.COM
I20260620 11:33:43.058532 9842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.051525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:38923 (local address 127.0.0.1:53672)
0620 11:33:43.051576 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.051587 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.051625 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.051770 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.051771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.051917 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:43.051944 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.051945 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.052654 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.052656 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.053154 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.053156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.053529 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.053529 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.053584 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.053935 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:43.053940 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:43.055431 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:43.057353 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.057354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.057354 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.057545 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.057794 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.057795 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.057796 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.057867 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.058165 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:43.058166 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:43.058217 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.058312 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.058424 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 11:33:43.058545 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.051601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53672 (local address 127.2.243.189:38923)
0620 11:33:43.051641 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.051643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.051651 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:33:43.051680 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.051681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.051697 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.051766 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.052028 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.052581 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.053235 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.053460 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.053532 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.055545 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.055547 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.055547 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.055556 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.057213 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.057642 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.057643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.057644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.057709 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.057953 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.057954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.057955 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.058088 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.058174 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.058354 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.058449 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:33:43.059058 9840 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 11:33:43 dist-test-slave-ctbw krb5kdc[9075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955223, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.188@KRBTEST.COM for oryx/127.2.243.188@KRBTEST.COM
I20260620 11:33:43.065670 9842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.059317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:38025 (local address 127.0.0.1:52006)
0620 11:33:43.059369 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.059381 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.059430 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.059570 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.059571 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.059717 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:43.059746 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.059747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.060460 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.060461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.060973 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.060975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.061413 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.061414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.061491 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.061820 (+ 329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:43.061825 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:43.063027 (+ 1202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:43.064650 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.064651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.064652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.064833 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.065053 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.065054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.065054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.065121 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.065418 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:43.065419 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:43.065456 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.065511 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.065580 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 11:33:43.065860 9843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.059407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52006 (local address 127.2.243.188:38025)
0620 11:33:43.059445 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.059447 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.059462 (+ 15us) server_negotiation.cc:407] Connection header received
0620 11:33:43.059487 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.059487 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.059499 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.059566 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.059833 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.060349 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.061076 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.061320 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.061415 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.063106 (+ 1691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.063108 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.063108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.063114 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.064493 (+ 1379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.064899 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.064900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.064900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.064955 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.065190 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.065191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.065191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.065349 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.065415 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.065568 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.065647 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:33:43.067623 9840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23: Bootstrap starting.
I20260620 11:33:43.068552 9840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:43.068837 9840 log.cc:824] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:43.069640 9840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23: No bootstrap required, opened a new log
I20260620 11:33:43.071162 9840 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } }
I20260620 11:33:43.071285 9840 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:43.071316 9840 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46d54b8fefdc47e7b1e30ffe0d1adc23, State: Initialized, Role: FOLLOWER
I20260620 11:33:43.071465 9840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } }
I20260620 11:33:43.071774 9840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:43.071733 9845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } } }
I20260620 11:33:43.071870 9845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 11:33:43.073467 9856 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:33:43.073544 9856 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:33:43.075994 9772 master_runner.cc:428] Detected that this master 46d54b8fefdc47e7b1e30ffe0d1adc23 is joining an existing cluster
I20260620 11:33:43.076045 9772 master_runner.cc:432] Initiating AddMaster RPC to add 127.2.243.188:38025
I20260620 11:33:43.084512 9843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.077603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52016 (local address 127.2.243.188:38025)
0620 11:33:43.077688 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.077690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.077801 (+ 111us) server_negotiation.cc:407] Connection header received
0620 11:33:43.077895 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.077896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.077906 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.077936 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.078452 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.079239 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.080330 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.080597 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.080635 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.081715 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.081716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.081716 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.081723 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.083261 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.083640 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.083641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.083642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.083714 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.083957 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.083958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.083958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.084077 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.084153 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.084295 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.084405 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 11:33:43.084604 9862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.077543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:38025 (local address 127.0.0.1:52016)
0620 11:33:43.077728 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.077740 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.077826 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.077998 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.077998 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.078291 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:43.078368 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.078371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.079320 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.079321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.080241 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.080243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.080718 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.080718 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.080773 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.081289 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:43.081294 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:43.081620 (+ 326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:43.083336 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.083337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.083337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.083554 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.083793 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.083794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.083794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.083882 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.084143 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:43.084144 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:43.084199 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.084275 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.084462 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":98,"thread_start_us":78,"threads_started":1}
I20260620 11:33:43.102464 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 9182
I20260620 11:33:43.103691 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.077801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53674 (local address 127.2.243.189:38923)
0620 11:33:43.077841 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.077843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.077986 (+ 143us) server_negotiation.cc:407] Connection header received
0620 11:33:43.078103 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.078103 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.078114 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.078145 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.078592 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.079393 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.080352 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.080647 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.080684 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.100508 (+ 19824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.100510 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.100511 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.100519 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.102211 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.102634 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.102636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.102636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.102684 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.103063 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.103063 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.103064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.103214 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.103256 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.103457 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.103549 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:33:43.104080 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.077805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:38923 (local address 127.0.0.1:53674)
0620 11:33:43.077931 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.077940 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.078022 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.078206 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.078207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.078445 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:43.078512 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.078513 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.079475 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.079475 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.080258 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.080259 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.080749 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.080749 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.080800 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.099945 (+ 19145us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:43.099950 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:43.100395 (+ 445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:43.102294 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.102295 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.102296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.102548 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.102747 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.102748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.102748 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.102799 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.103302 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:43.103303 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:43.103351 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.103777 (+ 426us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.103914 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":14,"thread_start_us":73,"threads_started":1}
I20260620 11:33:43.105562 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.077649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44062 (local address 127.2.243.190:41191)
0620 11:33:43.077722 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.077724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.077990 (+ 266us) server_negotiation.cc:407] Connection header received
0620 11:33:43.078112 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.078113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.078124 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.078163 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.078635 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.079413 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.080463 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.080755 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.080806 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.100193 (+ 19387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.100195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.100196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.100205 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.102031 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.102528 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.102530 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.102530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.102590 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.104905 (+ 2315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.104906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.104906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.105062 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.105125 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.105327 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.105421 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 11:33:43.107026 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.077547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.0.0.1:44062)
0620 11:33:43.077926 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.077936 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.078028 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.078230 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.078231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.078480 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:43.078541 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.078543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.079509 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.079509 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.080364 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.080366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.080883 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.080884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.080945 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.099501 (+ 18556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:43.099508 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:43.100024 (+ 516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:43.102142 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.102143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.102144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.102438 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.104760 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.104761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.104761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.104827 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.105151 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:43.105152 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:43.105210 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.106785 (+ 1575us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.106943 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":78,"threads_started":1}
I20260620 11:33:43.108734 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:42977
--local_ip_for_outbound_sockets=127.2.243.129
--tserver_master_addrs=127.2.243.190:41191
--webserver_port=43457
--webserver_interface=127.2.243.129
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.129.keytab
--principal=oryx/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:34177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.243.190:41191,127.2.243.189:38923
--tserver_master_addrs=127.2.243.190:41191,127.2.243.189:38923,127.2.243.188:38025 with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 11:33:43 dist-test-slave-ctbw krb5kdc[9075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955222, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.190@KRBTEST.COM for oryx/127.2.243.188@KRBTEST.COM
I20260620 11:33:43.119520 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.110425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:38025 (local address 127.0.0.1:52024)
0620 11:33:43.110488 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.110500 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.110540 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.110724 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.110725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.110943 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:43.110998 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.111000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.112038 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.112040 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.113021 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.113024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.113535 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.113536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.113619 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.114145 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:43.114151 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:43.115929 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:43.118078 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.118080 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.118081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.118329 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.118617 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.118619 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.118619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.118692 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.119091 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:43.119092 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:43.119151 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.119243 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.119369 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 11:33:43.119526 9843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.110532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52024 (local address 127.2.243.188:38025)
0620 11:33:43.110576 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.110578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.110586 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:33:43.110624 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.110624 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.110636 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.110708 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.111096 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.111923 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.113113 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.113443 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.113526 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.116031 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.116033 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.116033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.116042 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.117976 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.118426 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.118428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.118428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.118522 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.118821 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.118822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.118822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.119009 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.119086 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.119287 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.119377 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:33:43.120595 9119 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.2.243.188:38025: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23"
member_type: NON_VOTER
last_known_addr {
host: "127.2.243.188"
port: 38025
}
attrs {
promote: true
}
}
dest_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a"
cas_config_opid_index: 5
I20260620 11:33:43.120862 9119 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: NON_VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: true } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: NON_VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: true } }
I20260620 11:33:43.121206 9680 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 46d54b8fefdc47e7b1e30ffe0d1adc23 (127.2.243.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: NON_VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: true } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: NON_VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: true } } }
I20260620 11:33:43.122231 9680 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.2.243.188:38025
I20260620 11:33:43.122308 9680 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: NON_VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: true } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: NON_VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: true } } }
I20260620 11:33:43.122385 9680 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:43.123153 9814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:43.124171 9814 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [term 1 FOLLOWER]: Refusing update from remote peer a5a8f84236d84cfb9ecad30874a8dc5a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 11:33:43.124469 9679 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: NON_VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:43.124960 9772 master.cc:537] Master@127.2.243.188:38025 shutting down...
I20260620 11:33:43.126178 9814 raft_consensus.cc:858] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: NON_VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: NON_VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: true } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: NON_VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: true } }
W20260620 11:33:43.126294 9814 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: NON_VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: NON_VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: true } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: NON_VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: true } }
I20260620 11:33:43.126804 9814 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3aa17d267b434b8781609a0171af2bca (127.2.243.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: NON_VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: true } } }
I20260620 11:33:43.128073 9869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: NON_VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: true } } }
I20260620 11:33:43.128167 9869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 11:33:43.128201 9869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: NON_VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: true } } }
I20260620 11:33:43.128268 9845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5a8f84236d84cfb9ecad30874a8dc5a. Latest consensus state: current_term: 1 leader_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: NON_VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: true } } }
I20260620 11:33:43.128338 9845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 11:33:43.128262 9869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 11:33:43.129762 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.122607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:38923 (local address 127.0.0.1:53686)
0620 11:33:43.122660 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.122672 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.122744 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.123080 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.123081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.123083 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:43.123124 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.123125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.124123 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.124124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.128534 (+ 4410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.128539 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.129479 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.129481 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.129555 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.129581 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.129643 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.129687 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 11:33:43.129763 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.122814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53686 (local address 127.2.243.189:38923)
0620 11:33:43.122890 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.122892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.122899 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:33:43.122936 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.122937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.122961 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.123019 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:43.123197 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.123976 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.128662 (+ 4686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.129371 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.129465 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.129546 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.129646 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.129690 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 11:33:43.130295 9728 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 1 LEARNER]: Refusing update from remote peer a5a8f84236d84cfb9ecad30874a8dc5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 11:33:43.130517 9679 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: NON_VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 11:33:43.131168 9728 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 46d54b8fefdc47e7b1e30ffe0d1adc23 (127.2.243.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: NON_VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: true } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: NON_VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: true } } }
I20260620 11:33:43.132459 9757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5a8f84236d84cfb9ecad30874a8dc5a. Latest consensus state: current_term: 1 leader_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: NON_VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: true } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: NON_VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: true } } }
I20260620 11:33:43.132543 9757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [sys.catalog]: This master's current role is: LEARNER
I20260620 11:33:43.132579 9757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: NON_VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: true } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: NON_VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: true } } }
I20260620 11:33:43.132624 9757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [sys.catalog]: This master's current role is: LEARNER
I20260620 11:33:43.132656 9679 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a: attempting to promote NON_VOTER 3aa17d267b434b8781609a0171af2bca to VOTER
I20260620 11:33:43.132933 9679 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: NON_VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: true } }
I20260620 11:33:43.133555 9728 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 1 LEARNER]: Refusing update from remote peer a5a8f84236d84cfb9ecad30874a8dc5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 11:33:43.133774 9872 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260620 11:33:43.134014 9105 consensus_peers.cc:487] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a -> Peer 46d54b8fefdc47e7b1e30ffe0d1adc23 (127.2.243.188:38025): Couldn't send request to peer 46d54b8fefdc47e7b1e30ffe0d1adc23. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 11:33:43.134387 9680 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3aa17d267b434b8781609a0171af2bca (127.2.243.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: NON_VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: true } } }
I20260620 11:33:43.134542 9728 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3aa17d267b434b8781609a0171af2bca (127.2.243.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: NON_VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: true } } }
I20260620 11:33:43.135468 9866 mvcc.cc:205] Tried to move back new op lower bound from 7298888589667680256 to 7298888589576695808. Current Snapshot: MvccSnapshot[applied={T|T < 7298888589586997248}]
I20260620 11:33:43.135766 9871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: NON_VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: true } } }
I20260620 11:33:43.135850 9871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:33:43.135793 9680 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: NON_VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: true } } }
I20260620 11:33:43.135905 9680 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:43.135905 9872 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: NON_VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: true } } }
I20260620 11:33:43.136076 9872 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:43.136947 9772 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260620 11:33:43.137063 9772 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260620 11:33:43.137099 9772 tablet_replica.cc:330] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23: stopping tablet replica
I20260620 11:33:43.154253 9772 master.cc:552] Master@127.2.243.188:38025 shutdown complete.
I20260620 11:33:43.154317 9772 master_runner.cc:305] Clearing existing system tablet
I20260620 11:33:43.154449 9772 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:43.154498 9772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:43.155333 9772 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:33:43.156016 9878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:43.156201 9772 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:43.156284 9772 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:43.156315 9772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/wal
uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23"
format_stamp: "Formatted at 2026-06-20 11:33:43 on dist-test-slave-ctbw"
server_key: "013b2efca99e6ef716865dc6dfa6c057"
server_key_iv: "b1a374bef8f89c9871299d2b21414605"
server_key_version: "encryptionkey@0"
I20260620 11:33:43.157133 9772 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:43.158490 9772 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 11:33:43.158560 9772 log.cc:1196] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 11:33:43.158774 9772 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23: Deleting consensus metadata
I20260620 11:33:43.158969 9772 master_runner.cc:315] Copying system tablet from 127.2.243.190:41191
I20260620 11:33:43.159567 9772 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23: tablet copy: Beginning tablet copy session from remote peer at address 127.2.243.190:41191
I20260620 11:33:43.167408 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.160090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44074 (local address 127.2.243.190:41191)
0620 11:33:43.160138 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.160140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.160152 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:33:43.160181 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.160182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.160193 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.160272 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.160633 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.161479 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.162515 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.162832 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.162974 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.164170 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.164172 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.164173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.164181 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.166098 (+ 1917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.166484 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.166485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.166486 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.166563 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.166791 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.166792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.166792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.166993 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.167070 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.167225 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.167309 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 11:33:43.167490 9884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.159878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.0.0.1:44074)
0620 11:33:43.160038 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.160053 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.160122 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.160284 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.160285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.160483 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:43.160536 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.160538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.161566 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.161567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.162409 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.162412 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.162972 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.162973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.163031 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.163602 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:43.163607 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:43.164079 (+ 472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:43.166198 (+ 2119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.166199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.166200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.166420 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.166641 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.166642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.166642 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.166708 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.167068 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:43.167069 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:43.167119 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.167195 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.167354 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260620 11:33:43.168108 9140 tablet_copy_service.cc:140] P a5a8f84236d84cfb9ecad30874a8dc5a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 46d54b8fefdc47e7b1e30ffe0d1adc23 ({username='oryx', principal='oryx/127.2.243.188@KRBTEST.COM'} at 127.0.0.1:44074)
I20260620 11:33:43.168180 9140 tablet_copy_service.cc:161] P a5a8f84236d84cfb9ecad30874a8dc5a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 46d54b8fefdc47e7b1e30ffe0d1adc23 at {username='oryx', principal='oryx/127.2.243.188@KRBTEST.COM'} at 127.0.0.1:44074: session id = 46d54b8fefdc47e7b1e30ffe0d1adc23-00000000000000000000000000000000
I20260620 11:33:43.168778 9140 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 11:33:43.169314 9772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 11:33:43.170594 9772 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23: tablet copy: Starting download of 0 data blocks...
I20260620 11:33:43.170739 9772 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23: tablet copy: Starting download of 1 WAL segments...
I20260620 11:33:43.171623 9772 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 11:33:43.172627 9140 tablet_copy_service.cc:342] P a5a8f84236d84cfb9ecad30874a8dc5a: Request end of tablet copy session 46d54b8fefdc47e7b1e30ffe0d1adc23-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.2.243.188@KRBTEST.COM'} at 127.0.0.1:44074
I20260620 11:33:43.172683 9140 tablet_copy_service.cc:433] P a5a8f84236d84cfb9ecad30874a8dc5a: ending tablet copy session 46d54b8fefdc47e7b1e30ffe0d1adc23-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 46d54b8fefdc47e7b1e30ffe0d1adc23
I20260620 11:33:43.175549 9772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:43.176709 9885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:43.176805 9888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:43.177114 9772 server_base.cc:1034] running on GCE node
W20260620 11:33:43.179028 9886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:43.179239 9772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:43.179385 9772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:43.181383 9772 hybrid_clock.cc:648] HybridClock initialized: now 1781955223181360 us; error 40 us; skew 500 ppm
Jun 20 11:33:43 dist-test-slave-ctbw krb5kdc[9075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955223, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:43.183053 9772 init.cc:377] Logged in from keytab as oryx/127.2.243.188@KRBTEST.COM (short username oryx)
I20260620 11:33:43.183317 9772 webserver.cc:460] Webserver started at http://127.2.243.188:34743/ using document root <none> and password file <none>
I20260620 11:33:43.183442 9772 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:43.183478 9772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:43.184252 9772 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:43.184865 9895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:43.185057 9772 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:43.185113 9772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/wal
uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23"
format_stamp: "Formatted at 2026-06-20 11:33:43 on dist-test-slave-ctbw"
server_key: "013b2efca99e6ef716865dc6dfa6c057"
server_key_iv: "b1a374bef8f89c9871299d2b21414605"
server_key_version: "encryptionkey@0"
I20260620 11:33:43.185173 9772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:43.202080 9772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:43.204088 9772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:43.207487 9772 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.188:38025
I20260620 11:33:43.207628 9947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.188:38025 every 8 connection(s)
I20260620 11:33:43.207839 9772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/master-2/data/info.pb
I20260620 11:33:43.208810 9948 sys_catalog.cc:263] Verifying existing consensus state
I20260620 11:33:43.209438 9948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23: Bootstrap starting.
I20260620 11:33:43.212054 9948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:33:43.212345 9948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23: Bootstrap complete.
I20260620 11:33:43.212579 9948 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: NON_VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: true } }
I20260620 11:33:43.212672 9948 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 46d54b8fefdc47e7b1e30ffe0d1adc23, State: Initialized, Role: LEARNER
I20260620 11:33:43.212718 9948 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: NON_VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: true } }
I20260620 11:33:43.212956 9948 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:43.212910 9951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: NON_VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: true } } }
I20260620 11:33:43.213158 9951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [sys.catalog]: This master's current role is: LEARNER
I20260620 11:33:43.214673 9962 catalog_manager.cc:1259] Loaded cluster ID: 0cbb43cecc8540a98dd3fee84da7a872
I20260620 11:33:43.214726 9962 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23: loading cluster ID for follower catalog manager: success
I20260620 11:33:43.216020 9962 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23: acquiring CA information for follower catalog manager: success
I20260620 11:33:43.216338 9962 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 11:33:43.218221 9865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:43.218444 9865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:43.218492 9865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:43.219847 9865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:43.219913 9865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:43.219966 9865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:33:43.221571 9865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:34177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.129.keytab
--principal=oryx/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:42977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=43457
--tserver_master_addrs=127.2.243.190:41191,127.2.243.189:38923,127.2.243.188:38025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:43.221860 9865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:43.222116 9865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:43.228438 9968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:43.228448 9967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:43.228562 9970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:43.228670 9865 server_base.cc:1034] running on GCE node
I20260620 11:33:43.228921 9865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:43.229171 9865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:43.230333 9865 hybrid_clock.cc:648] HybridClock initialized: now 1781955223230318 us; error 36 us; skew 500 ppm
Jun 20 11:33:43 dist-test-slave-ctbw krb5kdc[9075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955223, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:43.233197 9865 init.cc:377] Logged in from keytab as oryx/127.2.243.129@KRBTEST.COM (short username oryx)
I20260620 11:33:43.233760 9865 webserver.cc:460] Webserver started at http://127.2.243.129:43457/ using document root <none> and password file <none>
I20260620 11:33:43.234091 9865 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:43.234180 9865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:43.236124 9865 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:43.236788 9977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:43.236991 9865 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:43.237084 9865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "355a1e4b20b64b58864cc1ebd763ea12"
format_stamp: "Formatted at 2026-06-20 11:33:42 on dist-test-slave-ctbw"
server_key: "324e277e909e2632743ef4b9ca68b4a2"
server_key_iv: "7bb3e7cd55573eab43cadc134e0b7f13"
server_key_version: "encryptionkey@0"
I20260620 11:33:43.237510 9865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:43.257789 9865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:43.260669 9865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:43.260828 9865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:43.261102 9865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:43.261468 9865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:43.261519 9865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:43.261554 9865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:43.261580 9865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:43.268921 9865 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:42977
I20260620 11:33:43.269164 10090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:42977 every 8 connection(s)
I20260620 11:33:43.269380 9865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:33:43.270305 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 9865
I20260620 11:33:43.270493 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 9316
Jun 20 11:33:43 dist-test-slave-ctbw krb5kdc[9075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955223, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM
I20260620 11:33:43.279088 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:41803
--local_ip_for_outbound_sockets=127.2.243.130
--tserver_master_addrs=127.2.243.190:41191
--webserver_port=40939
--webserver_interface=127.2.243.130
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.130.keytab
--principal=oryx/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:34177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.243.190:41191,127.2.243.189:38923
--tserver_master_addrs=127.2.243.190:41191,127.2.243.189:38923,127.2.243.188:38025 with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:33:43.280827 10095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.270169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.2.243.129:41881)
0620 11:33:43.270762 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.270772 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.271622 (+ 850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.271996 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.272008 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.272601 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:43.272796 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.272800 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.273516 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.273517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.274362 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.274364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.274753 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.274754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.274941 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.275615 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:43.275624 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:43.277054 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:43.279068 (+ 2014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.279076 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.279079 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.279446 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.279657 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.279657 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.279658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.279819 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.280128 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:43.280134 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:43.280300 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.280421 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.280583 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":3,"thread_start_us":100,"threads_started":1}
I20260620 11:33:43.281250 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.270513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:41881 (local address 127.2.243.190:41191)
0620 11:33:43.270574 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.270576 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.270820 (+ 244us) server_negotiation.cc:407] Connection header received
0620 11:33:43.271756 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.271758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.271768 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.271900 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.272899 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.273417 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.274470 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.274660 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.274745 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.277165 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.277166 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.277167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.277174 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.278976 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.279526 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.279527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.279528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.279589 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.279898 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.279899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.279900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.280047 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.280104 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.281077 (+ 973us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.281161 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 11:33:43.281595 10091 heartbeater.cc:347] Connected to a master server at 127.2.243.190:41191
I20260620 11:33:43.281693 10091 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:43.281903 10091 heartbeater.cc:511] Master 127.2.243.190:41191 requested a full tablet report, sending...
I20260620 11:33:43.282305 9119 ts_manager.cc:194] Re-registered known tserver with Master: 355a1e4b20b64b58864cc1ebd763ea12 (127.2.243.129:42977)
I20260620 11:33:43.282773 9119 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:41881
Jun 20 11:33:43 dist-test-slave-ctbw krb5kdc[9075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955223, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.189@KRBTEST.COM
Jun 20 11:33:43 dist-test-slave-ctbw krb5kdc[9075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955223, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.188@KRBTEST.COM
I20260620 11:33:43.288875 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.279297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:47249 (local address 127.2.243.189:38923)
0620 11:33:43.279339 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.279341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.279457 (+ 116us) server_negotiation.cc:407] Connection header received
0620 11:33:43.279565 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.279567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.279578 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.279610 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.280054 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.280778 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.281842 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.282118 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.282159 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.285359 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.285361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.285362 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.285370 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.287369 (+ 1999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.287994 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.287996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.287996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.288069 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.288285 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.288286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.288287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.288423 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.288508 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.288672 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.288762 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:33:43.288888 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.279117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:38923 (local address 127.2.243.129:47249)
0620 11:33:43.279396 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.279407 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.279482 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.279697 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.279698 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.279875 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:43.279964 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.279966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.280896 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.280897 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.281738 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.281741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.282236 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.282237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.282295 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.282806 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:43.282811 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:43.285255 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:43.287473 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.287474 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.287475 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.287665 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.288144 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.288145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.288145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.288216 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.288502 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:43.288503 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:43.288552 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.288647 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.288762 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":78,"threads_started":1}
I20260620 11:33:43.289247 10093 heartbeater.cc:347] Connected to a master server at 127.2.243.189:38923
I20260620 11:33:43.289309 10093 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:43.289402 10093 heartbeater.cc:511] Master 127.2.243.189:38923 requested a full tablet report, sending...
I20260620 11:33:43.289846 9718 ts_manager.cc:194] Registered new tserver with Master: 355a1e4b20b64b58864cc1ebd763ea12 (127.2.243.129:42977)
I20260620 11:33:43.290771 10097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.270500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:57833 (local address 127.2.243.188:38025)
0620 11:33:43.278839 (+ 8339us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.278842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.278853 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:33:43.278916 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.278917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.278934 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.279300 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.279874 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.281625 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.283170 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.283561 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.283934 (+ 373us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.286690 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.286691 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.286692 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.286701 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.288769 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.289718 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.289719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.289720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.289787 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.290056 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.290057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.290058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.290226 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.290314 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.290512 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.290622 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8305,"thread_start_us":56,"threads_started":1}
I20260620 11:33:43.290802 10096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.270169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:38025 (local address 127.2.243.129:57833)
0620 11:33:43.270385 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.270446 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.271794 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.279054 (+ 7260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.279056 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.279677 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:43.279771 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.279773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.281816 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.281818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.283064 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.283067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.283670 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.283672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.283747 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.284196 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:43.284202 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:43.286229 (+ 2027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:43.288879 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.288880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.288881 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.289625 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.289872 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.289873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.289874 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.289959 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.290306 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:43.290306 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:43.290387 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.290453 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.290543 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"spinlock_wait_cycles":7936,"thread_start_us":100,"threads_started":1}
I20260620 11:33:43.291314 10092 heartbeater.cc:347] Connected to a master server at 127.2.243.188:38025
I20260620 11:33:43.291380 10092 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:43.291436 10092 heartbeater.cc:511] Master 127.2.243.188:38025 requested a full tablet report, sending...
I20260620 11:33:43.291780 9912 ts_manager.cc:194] Registered new tserver with Master: 355a1e4b20b64b58864cc1ebd763ea12 (127.2.243.129:42977)
W20260620 11:33:43.369489 10099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:43.369690 10099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:43.369714 10099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:43.370906 10099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:43.370966 10099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:43.370996 10099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:33:43.372473 10099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:34177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.130.keytab
--principal=oryx/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:41803
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=40939
--tserver_master_addrs=127.2.243.190:41191,127.2.243.189:38923,127.2.243.188:38025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:43.372709 10099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:43.372967 10099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:43.377887 10105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:43.377868 10104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:43.377925 10107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:43.378137 10099 server_base.cc:1034] running on GCE node
I20260620 11:33:43.378336 10099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:43.378608 10099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:43.379792 10099 hybrid_clock.cc:648] HybridClock initialized: now 1781955223379792 us; error 44 us; skew 500 ppm
Jun 20 11:33:43 dist-test-slave-ctbw krb5kdc[9075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955223, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:43.382602 10099 init.cc:377] Logged in from keytab as oryx/127.2.243.130@KRBTEST.COM (short username oryx)
I20260620 11:33:43.383133 10099 webserver.cc:460] Webserver started at http://127.2.243.130:40939/ using document root <none> and password file <none>
I20260620 11:33:43.383410 10099 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:43.383499 10099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:43.384944 10099 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 11:33:43.385609 10114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:43.385819 10099 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:43.385918 10099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "a97ab72ec734489c8cff99acbc81b178"
format_stamp: "Formatted at 2026-06-20 11:33:42 on dist-test-slave-ctbw"
server_key: "de2fcc80fb6723fbcd2d176296df029f"
server_key_iv: "8c6f64736ad18282e6a90d4c1a5ff2e1"
server_key_version: "encryptionkey@0"
I20260620 11:33:43.386322 10099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:43.399443 10099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:43.401571 10099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:43.401744 10099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:43.402004 10099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:43.402367 10099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:43.402424 10099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:43.402467 10099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:43.402484 10099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:43.409117 10099 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:41803
I20260620 11:33:43.409209 10227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:41803 every 8 connection(s)
I20260620 11:33:43.409560 10099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:33:43.415696 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 10099
I20260620 11:33:43.415872 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 9450
Jun 20 11:33:43 dist-test-slave-ctbw krb5kdc[9075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955223, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.188@KRBTEST.COM
Jun 20 11:33:43 dist-test-slave-ctbw krb5kdc[9075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955223, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.189@KRBTEST.COM
Jun 20 11:33:43 dist-test-slave-ctbw krb5kdc[9075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955223, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM
I20260620 11:33:43.421844 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.410690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:34913 (local address 127.2.243.189:38923)
0620 11:33:43.410753 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.410755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.410882 (+ 127us) server_negotiation.cc:407] Connection header received
0620 11:33:43.411882 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.411883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.411896 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.411928 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.412744 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.413577 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.414725 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.415079 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.415116 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.418218 (+ 3102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.418220 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.418220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.418228 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.420167 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.420707 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.420709 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.420709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.420767 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.421057 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.421058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.421059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.421201 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.421245 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.421634 (+ 389us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.421720 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 11:33:43.422070 10097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.410405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:56823 (local address 127.2.243.188:38025)
0620 11:33:43.410453 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.410455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.410809 (+ 354us) server_negotiation.cc:407] Connection header received
0620 11:33:43.411848 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.411849 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.411861 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.411908 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.412719 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.413339 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.414571 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.414854 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.414961 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.418771 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.418773 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.418773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.418780 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.420787 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.421172 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.421173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.421174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.421227 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.421445 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.421446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.421446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.421590 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.421631 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.421827 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.421932 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:33:43.422523 10234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.410332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:38025 (local address 127.2.243.130:56823)
0620 11:33:43.410734 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.410744 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.411753 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.411977 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.411981 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.412374 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:43.412630 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.412635 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.413444 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.413445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.414473 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.414475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.414969 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.414970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.415105 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.415613 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:43.415624 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:43.418671 (+ 3047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:43.420875 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.420876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.420877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.421096 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.421292 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.421293 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.421293 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.421381 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.421675 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:43.421676 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:43.421720 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.422152 (+ 432us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.422296 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":50,"thread_start_us":90,"threads_started":1}
I20260620 11:33:43.422531 10232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.410626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:38923 (local address 127.2.243.130:34913)
0620 11:33:43.410807 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.410815 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.411765 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.411997 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.411998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.412396 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:43.412631 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.412637 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.413689 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.413689 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.414625 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.414626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.415189 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.415190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.415274 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.415770 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:43.415775 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:43.418099 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:43.420262 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.420269 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.420272 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.420612 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.420839 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.420840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.420840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.420973 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.421292 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:43.421297 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:43.421480 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.421953 (+ 473us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.422233 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":34,"thread_start_us":64,"threads_started":1}
I20260620 11:33:43.422788 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.410322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:54735 (local address 127.2.243.190:41191)
0620 11:33:43.410380 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.410382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.410635 (+ 253us) server_negotiation.cc:407] Connection header received
0620 11:33:43.411872 (+ 1237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.411874 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.411895 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.411929 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.412743 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.413551 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.414738 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.415042 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.415103 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.419284 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.419285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.419286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.419294 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.421200 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.421574 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.421576 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.421576 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.421634 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.421825 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.421826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.421826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.421964 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.422008 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.422604 (+ 596us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.422684 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 11:33:43.423390 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.410332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.2.243.130:54735)
0620 11:33:43.410510 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.410555 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.411750 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.411997 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.411998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.412383 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:43.412631 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.412637 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.413668 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.413669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.414618 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.414620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.415189 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.415190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.415820 (+ 630us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.416251 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:43.416257 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:43.419173 (+ 2916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:43.421281 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.421282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.421283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.421485 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.421706 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.421706 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.421706 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.421759 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.422440 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:43.422440 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:43.422480 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.423169 (+ 689us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.423267 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":109,"thread_start_us":89,"threads_started":1}
I20260620 11:33:43.423995 10230 heartbeater.cc:347] Connected to a master server at 127.2.243.189:38923
I20260620 11:33:43.424083 10230 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:43.424263 10230 heartbeater.cc:511] Master 127.2.243.189:38923 requested a full tablet report, sending...
I20260620 11:33:43.424340 10229 heartbeater.cc:347] Connected to a master server at 127.2.243.188:38025
I20260620 11:33:43.424393 10229 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:43.424463 10229 heartbeater.cc:511] Master 127.2.243.188:38025 requested a full tablet report, sending...
I20260620 11:33:43.424576 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:46759
--local_ip_for_outbound_sockets=127.2.243.131
--tserver_master_addrs=127.2.243.190:41191
--webserver_port=37007
--webserver_interface=127.2.243.131
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.131.keytab
--principal=oryx/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:34177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.2.243.190:41191,127.2.243.189:38923
--tserver_master_addrs=127.2.243.190:41191,127.2.243.189:38923,127.2.243.188:38025 with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:33:43.424599 10228 heartbeater.cc:347] Connected to a master server at 127.2.243.190:41191
I20260620 11:33:43.424647 10228 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:43.424716 10228 heartbeater.cc:511] Master 127.2.243.190:41191 requested a full tablet report, sending...
I20260620 11:33:43.424767 9912 ts_manager.cc:194] Registered new tserver with Master: a97ab72ec734489c8cff99acbc81b178 (127.2.243.130:41803)
I20260620 11:33:43.424963 9119 ts_manager.cc:194] Re-registered known tserver with Master: a97ab72ec734489c8cff99acbc81b178 (127.2.243.130:41803)
I20260620 11:33:43.425056 9718 ts_manager.cc:194] Registered new tserver with Master: a97ab72ec734489c8cff99acbc81b178 (127.2.243.130:41803)
I20260620 11:33:43.425402 9119 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:54735
W20260620 11:33:43.506796 10235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:43.507109 10235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:43.507149 10235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:43.508431 10235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:43.508503 10235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:43.508550 10235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:33:43.510149 10235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:34177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.131.keytab
--principal=oryx/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:46759
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=37007
--tserver_master_addrs=127.2.243.190:41191,127.2.243.189:38923,127.2.243.188:38025
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:43.510437 10235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:43.510673 10235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:43.516058 10243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:43.516050 10241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:43.516355 10235 server_base.cc:1034] running on GCE node
W20260620 11:33:43.516076 10240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:43.516605 10235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:43.516867 10235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:43.518033 10235 hybrid_clock.cc:648] HybridClock initialized: now 1781955223518027 us; error 39 us; skew 500 ppm
Jun 20 11:33:43 dist-test-slave-ctbw krb5kdc[9075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955223, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:43.520617 10235 init.cc:377] Logged in from keytab as oryx/127.2.243.131@KRBTEST.COM (short username oryx)
I20260620 11:33:43.521108 10235 webserver.cc:460] Webserver started at http://127.2.243.131:37007/ using document root <none> and password file <none>
I20260620 11:33:43.521342 10235 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:43.521422 10235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:43.522748 10235 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:43.523470 10250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:43.523644 10235 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 11:33:43.523741 10235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "c3ec40c3f3a44f958349012a3bda33e1"
format_stamp: "Formatted at 2026-06-20 11:33:42 on dist-test-slave-ctbw"
server_key: "57c6ba798ac4047d60ec18c186a3fbaa"
server_key_iv: "67462b5dfaf1e23f5ccae987ed47e64b"
server_key_version: "encryptionkey@0"
I20260620 11:33:43.524096 10235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:43.535514 10235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:43.537536 10235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:43.537695 10235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:43.537940 10235 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:43.538300 10235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:43.538348 10235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:43.538388 10235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:43.538415 10235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:43.544569 10235 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:46759
I20260620 11:33:43.544632 10363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:46759 every 8 connection(s)
I20260620 11:33:43.545007 10235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:33:43.549625 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 10235
Jun 20 11:33:43 dist-test-slave-ctbw krb5kdc[9075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955223, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM
Jun 20 11:33:43 dist-test-slave-ctbw krb5kdc[9075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955223, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.189@KRBTEST.COM
Jun 20 11:33:43 dist-test-slave-ctbw krb5kdc[9075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955223, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.188@KRBTEST.COM
I20260620 11:33:43.557611 10370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.545838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:38923 (local address 127.2.243.131:55407)
0620 11:33:43.546258 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.546269 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.546901 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.547134 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.547137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.547600 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:43.547855 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.547861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.548814 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.548815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.549787 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.549789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.550320 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.550321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.550392 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.551075 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:43.551081 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:43.553894 (+ 2813us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:43.555856 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.555862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.555865 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.556184 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.556402 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.556403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.556403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.556552 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.556860 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:43.556867 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:43.557051 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.557176 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.557358 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":68,"thread_start_us":65,"threads_started":1}
I20260620 11:33:43.557757 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.546064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.2.243.131:58913)
0620 11:33:43.546477 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.546487 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.546898 (+ 411us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.547533 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.547534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.547692 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:43.547854 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.547860 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.548576 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.548577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.549387 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.549389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.549977 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.549978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.550120 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.550802 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:43.550812 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:43.553423 (+ 2611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:43.556405 (+ 2982us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.556406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.556406 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.556913 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.557129 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.557130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.557130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.557186 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.557473 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:43.557473 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:43.557512 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.557569 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.557662 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":55,"threads_started":1}
I20260620 11:33:43.558022 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.546199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:58913 (local address 127.2.243.190:41191)
0620 11:33:43.546243 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.546245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.546542 (+ 297us) server_negotiation.cc:407] Connection header received
0620 11:33:43.547021 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.547023 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.547034 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.547067 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.547965 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.548473 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.549543 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.549830 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.549879 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.553516 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.553517 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.553518 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.553525 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.556038 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.556988 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.556989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.556989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.557048 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.557249 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.557250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.557250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.557373 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.557446 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.557852 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.557933 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 11:33:43.558089 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.546208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:55407 (local address 127.2.243.189:38923)
0620 11:33:43.546250 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.546252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.546355 (+ 103us) server_negotiation.cc:407] Connection header received
0620 11:33:43.547001 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.547002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.547015 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.547073 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.547974 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.548715 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.549897 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.550193 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.550235 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.553984 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.553985 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.553986 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.553995 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.555763 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.556265 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.556266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.556267 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.556328 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.556635 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.556636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.556636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.556771 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.556831 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.557923 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.557995 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:33:43.558413 10366 heartbeater.cc:347] Connected to a master server at 127.2.243.189:38923
I20260620 11:33:43.558519 10366 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:43.558719 10366 heartbeater.cc:511] Master 127.2.243.189:38923 requested a full tablet report, sending...
I20260620 11:33:43.559051 10097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.546069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:55011 (local address 127.2.243.188:38025)
0620 11:33:43.546146 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.546148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.546359 (+ 211us) server_negotiation.cc:407] Connection header received
0620 11:33:43.547122 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.547124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.547135 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.547194 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.547973 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.548725 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.549902 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.550204 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.550241 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.555287 (+ 5046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.555288 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.555288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.555295 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.556995 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.557370 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.557371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.557371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.557442 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.557657 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.557658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.557658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.557782 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.557823 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.558855 (+ 1032us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.558959 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 11:33:43.559088 10364 heartbeater.cc:347] Connected to a master server at 127.2.243.190:41191
I20260620 11:33:43.559131 10364 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:43.559185 10364 heartbeater.cc:511] Master 127.2.243.190:41191 requested a full tablet report, sending...
I20260620 11:33:43.559208 9718 ts_manager.cc:194] Registered new tserver with Master: c3ec40c3f3a44f958349012a3bda33e1 (127.2.243.131:46759)
I20260620 11:33:43.559397 9119 ts_manager.cc:194] Re-registered known tserver with Master: c3ec40c3f3a44f958349012a3bda33e1 (127.2.243.131:46759)
I20260620 11:33:43.559667 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.545762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:38025 (local address 127.2.243.131:55011)
0620 11:33:43.546175 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.546269 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.546885 (+ 616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.547219 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.547219 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.547618 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:43.547855 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.547861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.548814 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.548815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.549817 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.549819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.550320 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.550321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.550963 (+ 642us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.551380 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:43.551396 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:43.555194 (+ 3798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:43.557075 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.557076 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.557076 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.557284 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.557523 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:43.557524 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:43.557525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:43.557585 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:43.558640 (+ 1055us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:43.558641 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:43.558686 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.559385 (+ 699us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.559546 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":34,"thread_start_us":90,"threads_started":1}
I20260620 11:33:43.559867 9119 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:58913
I20260620 11:33:43.560298 10365 heartbeater.cc:347] Connected to a master server at 127.2.243.188:38025
I20260620 11:33:43.560611 10365 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:43.560700 10365 heartbeater.cc:511] Master 127.2.243.188:38025 requested a full tablet report, sending...
I20260620 11:33:43.560994 9912 ts_manager.cc:194] Registered new tserver with Master: c3ec40c3f3a44f958349012a3bda33e1 (127.2.243.131:46759)
I20260620 11:33:43.561203 10377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.551043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52026 (local address 127.2.243.188:38025)
0620 11:33:43.551134 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.551136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.551754 (+ 618us) server_negotiation.cc:407] Connection header received
0620 11:33:43.551860 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.551861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.551875 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.551930 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.552686 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.553293 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.555990 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.556292 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.556335 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.558013 (+ 1678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.558015 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.558015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.558022 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.559642 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.560057 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.560058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.560059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.560118 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.560308 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.560309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.560310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.560447 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.560536 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.561032 (+ 496us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.561112 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":47,"threads_started":1}
I20260620 11:33:43.563313 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.555098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:38025 (local address 127.0.0.1:52034)
0620 11:33:43.555145 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.555154 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.555196 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.556863 (+ 1667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.556864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.556866 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:43.556929 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.556930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.560832 (+ 3902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.560834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.561872 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.561875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.563075 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.563076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.563138 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.563142 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.563203 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.563247 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 11:33:43.563336 10381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.554990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52034 (local address 127.2.243.188:38025)
0620 11:33:43.556671 (+ 1681us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.556673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.556680 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:33:43.556700 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.556701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.556729 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.558260 (+ 1531us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:43.558297 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.560770 (+ 2473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.561993 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.562988 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.563066 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.563144 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.563210 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.563252 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1656,"thread_start_us":55,"threads_started":1}
I20260620 11:33:43.563990 9922 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 11:33:43.564123 9951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5a8f84236d84cfb9ecad30874a8dc5a. Latest consensus state: current_term: 1 leader_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: NON_VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: true } } }
I20260620 11:33:43.564221 9951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [sys.catalog]: This master's current role is: LEARNER
I20260620 11:33:43.564543 10375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.550804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44078 (local address 127.2.243.190:41191)
0620 11:33:43.550966 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.550968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.551939 (+ 971us) server_negotiation.cc:407] Connection header received
0620 11:33:43.552038 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.552039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.552051 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.552094 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.552756 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.553481 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.559693 (+ 6212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.560005 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.560045 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.561539 (+ 1494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.561540 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.561541 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.561547 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.563252 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.563673 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.563674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.563674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.563731 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.563938 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.563938 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.563939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.564083 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.564174 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.564356 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.564443 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
I20260620 11:33:43.566244 9119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:33:43.568562 9119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:33:43.569370 10380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.551963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53700 (local address 127.2.243.189:38923)
0620 11:33:43.552064 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.552066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.553515 (+ 1449us) server_negotiation.cc:407] Connection header received
0620 11:33:43.553539 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.553540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.553553 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.553609 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.562365 (+ 8756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.563002 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.564955 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.565273 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.565347 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.566607 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.566608 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.566609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.566616 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.568157 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.568541 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.568542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.568543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.568604 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.568815 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.568816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.568816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.568949 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.569057 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.569173 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.569268 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":58,"threads_started":1}
I20260620 11:33:43.569336 9876 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: NON_VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: true } } }
I20260620 11:33:43.569422 9876 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:43.570288 9680 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a: attempting to promote NON_VOTER 46d54b8fefdc47e7b1e30ffe0d1adc23 to VOTER
I20260620 11:33:43.570488 9680 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: false } }
I20260620 11:33:43.570907 9728 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 1 FOLLOWER]: Refusing update from remote peer a5a8f84236d84cfb9ecad30874a8dc5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 11:33:43.570930 9922 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [term 1 LEARNER]: Refusing update from remote peer a5a8f84236d84cfb9ecad30874a8dc5a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 11:33:43.571118 9872 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 11:33:43.571187 9876 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 11:33:43.571825 9680 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 46d54b8fefdc47e7b1e30ffe0d1adc23 (127.2.243.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: false } } }
I20260620 11:33:43.571956 9728 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 46d54b8fefdc47e7b1e30ffe0d1adc23 (127.2.243.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: false } } }
I20260620 11:33:43.572588 9680 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: false } } }
I20260620 11:33:43.572647 9876 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: false } } }
I20260620 11:33:43.572732 9876 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:43.572686 9872 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: false } } }
I20260620 11:33:43.572762 9872 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:43.572722 9871 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: false } } }
I20260620 11:33:43.572814 9871 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:33:43.572878 9922 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 46d54b8fefdc47e7b1e30ffe0d1adc23 (127.2.243.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: false } } }
I20260620 11:33:43.573695 9951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: false } } }
I20260620 11:33:43.572685 9680 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:43.573786 9951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:33:43.580134 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.575943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:41803 (local address 127.0.0.1:60552)
0620 11:33:43.576249 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.576259 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.576308 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.576733 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.576734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.576736 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:43.576782 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.576784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.577727 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.577728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.578768 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.578771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.579742 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.579743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.579807 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.579812 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.579840 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.579876 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":73,"threads_started":1}
I20260620 11:33:43.580605 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.576326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60552 (local address 127.2.243.130:41803)
0620 11:33:43.576552 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.576556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.576566 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:33:43.576604 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.576607 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.576623 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.576853 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:43.576938 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.577641 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.578856 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.579650 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.580349 (+ 699us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.580436 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.580464 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.580511 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":140,"threads_started":1}
I20260620 11:33:43.581380 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.575932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:46759 (local address 127.0.0.1:58452)
0620 11:33:43.575978 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.575989 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.576028 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.576405 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.576406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.576408 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:43.576456 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.576457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.577765 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.577766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.579898 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.580029 (+ 131us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.581141 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.581142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.581214 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.581218 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.581250 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.581296 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":1011}
I20260620 11:33:43.581396 10162 tablet_service.cc:1467] Processing CreateTablet for tablet 77a2b63c144b4e588e8a9a4195fafbeb (DEFAULT_TABLE table=test-table [id=099a80d0986b42e6bceb5347c6b01559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:43.581808 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.576052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58452 (local address 127.2.243.131:46759)
0620 11:33:43.576197 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.576202 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.576212 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:33:43.576256 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.576258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.576296 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.576533 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:43.576610 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.577671 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.580156 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.581025 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.581549 (+ 524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.581637 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.581665 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.581715 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260620 11:33:43.581828 10162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a2b63c144b4e588e8a9a4195fafbeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:43.582442 10298 tablet_service.cc:1467] Processing CreateTablet for tablet 77a2b63c144b4e588e8a9a4195fafbeb (DEFAULT_TABLE table=test-table [id=099a80d0986b42e6bceb5347c6b01559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:43.582737 10298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a2b63c144b4e588e8a9a4195fafbeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:43.585824 10394 tablet_bootstrap.cc:492] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178: Bootstrap starting.
I20260620 11:33:43.586483 10393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.576383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49908 (local address 127.2.243.129:42977)
0620 11:33:43.576658 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.576661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.576769 (+ 108us) server_negotiation.cc:407] Connection header received
0620 11:33:43.576849 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.576851 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.576867 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.576938 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:43.577045 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.578051 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.579479 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.580293 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.580354 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.580433 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.586333 (+ 5900us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.586394 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":94,"threads_started":1}
I20260620 11:33:43.586591 10394 tablet_bootstrap.cc:654] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:43.586602 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.576111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:42977 (local address 127.0.0.1:49908)
0620 11:33:43.576709 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.576719 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.576772 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.576948 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.576949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.576950 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:43.576988 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.576989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.578141 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.578142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.579324 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.579327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.580467 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.580469 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.580524 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.580529 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.580593 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.580635 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":75,"threads_started":1}
I20260620 11:33:43.586994 10394 log.cc:824] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:43.587280 10395 tablet_bootstrap.cc:492] T 77a2b63c144b4e588e8a9a4195fafbeb P c3ec40c3f3a44f958349012a3bda33e1: Bootstrap starting.
I20260620 11:33:43.587287 10025 tablet_service.cc:1467] Processing CreateTablet for tablet 77a2b63c144b4e588e8a9a4195fafbeb (DEFAULT_TABLE table=test-table [id=099a80d0986b42e6bceb5347c6b01559]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:43.587622 10025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a2b63c144b4e588e8a9a4195fafbeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:43.587960 10395 tablet_bootstrap.cc:654] T 77a2b63c144b4e588e8a9a4195fafbeb P c3ec40c3f3a44f958349012a3bda33e1: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:43.588305 10395 log.cc:824] T 77a2b63c144b4e588e8a9a4195fafbeb P c3ec40c3f3a44f958349012a3bda33e1: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:43.588549 10394 tablet_bootstrap.cc:492] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178: No bootstrap required, opened a new log
I20260620 11:33:43.588613 10394 ts_tablet_manager.cc:1397] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 11:33:43.590669 10394 raft_consensus.cc:348] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 } } peers { permanent_uuid: "a97ab72ec734489c8cff99acbc81b178" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 41803 } } peers { permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 } }
I20260620 11:33:43.590804 10394 raft_consensus.cc:374] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:43.590827 10394 raft_consensus.cc:729] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97ab72ec734489c8cff99acbc81b178, State: Initialized, Role: FOLLOWER
I20260620 11:33:43.591215 10398 tablet_bootstrap.cc:492] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12: Bootstrap starting.
I20260620 11:33:43.591320 10394 consensus_queue.cc:260] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 } } peers { permanent_uuid: "a97ab72ec734489c8cff99acbc81b178" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 41803 } } peers { permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 } }
I20260620 11:33:43.591638 10228 heartbeater.cc:503] Master 127.2.243.190:41191 was elected leader, sending a full tablet report...
I20260620 11:33:43.591763 10394 ts_tablet_manager.cc:1428] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:33:43.591845 10398 tablet_bootstrap.cc:654] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:43.592190 10398 log.cc:824] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:43.592871 10398 tablet_bootstrap.cc:492] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12: No bootstrap required, opened a new log
I20260620 11:33:43.592941 10398 ts_tablet_manager.cc:1397] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:33:43.593096 10395 tablet_bootstrap.cc:492] T 77a2b63c144b4e588e8a9a4195fafbeb P c3ec40c3f3a44f958349012a3bda33e1: No bootstrap required, opened a new log
I20260620 11:33:43.593179 10395 ts_tablet_manager.cc:1397] T 77a2b63c144b4e588e8a9a4195fafbeb P c3ec40c3f3a44f958349012a3bda33e1: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260620 11:33:43.595121 10398 raft_consensus.cc:348] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 } } peers { permanent_uuid: "a97ab72ec734489c8cff99acbc81b178" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 41803 } } peers { permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 } }
I20260620 11:33:43.595265 10398 raft_consensus.cc:374] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:43.595306 10398 raft_consensus.cc:729] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 355a1e4b20b64b58864cc1ebd763ea12, State: Initialized, Role: FOLLOWER
I20260620 11:33:43.595427 10398 consensus_queue.cc:260] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 } } peers { permanent_uuid: "a97ab72ec734489c8cff99acbc81b178" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 41803 } } peers { permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 } }
I20260620 11:33:43.595456 10395 raft_consensus.cc:348] T 77a2b63c144b4e588e8a9a4195fafbeb P c3ec40c3f3a44f958349012a3bda33e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 } } peers { permanent_uuid: "a97ab72ec734489c8cff99acbc81b178" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 41803 } } peers { permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 } }
I20260620 11:33:43.595582 10395 raft_consensus.cc:374] T 77a2b63c144b4e588e8a9a4195fafbeb P c3ec40c3f3a44f958349012a3bda33e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:43.595654 10395 raft_consensus.cc:729] T 77a2b63c144b4e588e8a9a4195fafbeb P c3ec40c3f3a44f958349012a3bda33e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3ec40c3f3a44f958349012a3bda33e1, State: Initialized, Role: FOLLOWER
I20260620 11:33:43.595680 10398 ts_tablet_manager.cc:1428] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:33:43.595747 10091 heartbeater.cc:503] Master 127.2.243.190:41191 was elected leader, sending a full tablet report...
I20260620 11:33:43.595777 10395 consensus_queue.cc:260] T 77a2b63c144b4e588e8a9a4195fafbeb P c3ec40c3f3a44f958349012a3bda33e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 } } peers { permanent_uuid: "a97ab72ec734489c8cff99acbc81b178" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 41803 } } peers { permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 } }
I20260620 11:33:43.596062 10395 ts_tablet_manager.cc:1428] T 77a2b63c144b4e588e8a9a4195fafbeb P c3ec40c3f3a44f958349012a3bda33e1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:33:43.596297 10364 heartbeater.cc:503] Master 127.2.243.190:41191 was elected leader, sending a full tablet report...
I20260620 11:33:43.629554 10399 raft_consensus.cc:482] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:33:43.629673 10399 raft_consensus.cc:504] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 } } peers { permanent_uuid: "a97ab72ec734489c8cff99acbc81b178" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 41803 } } peers { permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 } }
I20260620 11:33:43.630064 10399 leader_election.cc:288] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3ec40c3f3a44f958349012a3bda33e1 (127.2.243.131:46759), 355a1e4b20b64b58864cc1ebd763ea12 (127.2.243.129:42977)
I20260620 11:33:43.631455 10401 raft_consensus.cc:482] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:33:43.631559 10401 raft_consensus.cc:504] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 } } peers { permanent_uuid: "a97ab72ec734489c8cff99acbc81b178" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 41803 } } peers { permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 } }
I20260620 11:33:43.631870 10401 leader_election.cc:288] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3ec40c3f3a44f958349012a3bda33e1 (127.2.243.131:46759), a97ab72ec734489c8cff99acbc81b178 (127.2.243.130:41803)
I20260620 11:33:43.633574 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.630352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:46759 (local address 127.2.243.130:40253)
0620 11:33:43.630430 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.630443 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.630488 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.630646 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.630647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.630648 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:43.630725 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.630726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.631445 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.631446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.632621 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.632623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.633357 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.633358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.633434 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.633436 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.633480 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.633507 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"mutex_wait_us":53}
I20260620 11:33:43.633730 10393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.630415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:50195 (local address 127.2.243.129:42977)
0620 11:33:43.630457 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.630460 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.630597 (+ 137us) server_negotiation.cc:407] Connection header received
0620 11:33:43.630648 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.630649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.630658 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.630738 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:43.630907 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.631448 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.632653 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.633197 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.633265 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.633274 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.633610 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.633656 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:33:43.633584 10232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.630352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:42977 (local address 127.2.243.130:50195)
0620 11:33:43.630499 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.630510 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.630588 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.630743 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.630744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.630746 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:43.630812 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.630816 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.631533 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.631534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.632585 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.632588 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.633307 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.633308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.633394 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.633397 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.633445 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.633492 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":101}
I20260620 11:33:43.633846 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.630415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:40253 (local address 127.2.243.131:46759)
0620 11:33:43.630465 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.630467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.630477 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:33:43.630572 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.630573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.630581 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.630644 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:43.630796 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.631335 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.632704 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.633271 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.633327 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.633333 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.633726 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.633769 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 11:33:43.634104 10318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a2b63c144b4e588e8a9a4195fafbeb" candidate_uuid: "a97ab72ec734489c8cff99acbc81b178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3ec40c3f3a44f958349012a3bda33e1" is_pre_election: true
I20260620 11:33:43.634104 10045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a2b63c144b4e588e8a9a4195fafbeb" candidate_uuid: "a97ab72ec734489c8cff99acbc81b178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "355a1e4b20b64b58864cc1ebd763ea12" is_pre_election: true
I20260620 11:33:43.634205 10045 raft_consensus.cc:2454] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97ab72ec734489c8cff99acbc81b178 in term 0.
I20260620 11:33:43.634230 10318 raft_consensus.cc:2454] T 77a2b63c144b4e588e8a9a4195fafbeb P c3ec40c3f3a44f958349012a3bda33e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97ab72ec734489c8cff99acbc81b178 in term 0.
I20260620 11:33:43.634416 10118 leader_election.cc:302] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97ab72ec734489c8cff99acbc81b178, c3ec40c3f3a44f958349012a3bda33e1; no voters:
I20260620 11:33:43.634588 10399 raft_consensus.cc:2781] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:33:43.634619 10399 raft_consensus.cc:482] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:33:43.634634 10399 raft_consensus.cc:3037] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:43.635496 10399 raft_consensus.cc:504] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 } } peers { permanent_uuid: "a97ab72ec734489c8cff99acbc81b178" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 41803 } } peers { permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 } }
I20260620 11:33:43.635627 10399 leader_election.cc:288] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 [CANDIDATE]: Term 1 election: Requested vote from peers c3ec40c3f3a44f958349012a3bda33e1 (127.2.243.131:46759), 355a1e4b20b64b58864cc1ebd763ea12 (127.2.243.129:42977)
I20260620 11:33:43.635816 10045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a2b63c144b4e588e8a9a4195fafbeb" candidate_uuid: "a97ab72ec734489c8cff99acbc81b178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "355a1e4b20b64b58864cc1ebd763ea12"
I20260620 11:33:43.635910 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.632696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:54023 (local address 127.2.243.130:41803)
0620 11:33:43.632734 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.632736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.632742 (+ 6us) server_negotiation.cc:407] Connection header received
0620 11:33:43.632760 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.632761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.632769 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.632814 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:43.632987 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.633643 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.634999 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.635525 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.635621 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.635629 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.635814 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.635847 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:33:43.635892 10098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.632450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:41803 (local address 127.2.243.129:54023)
0620 11:33:43.632518 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.632527 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.632557 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.632862 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.632863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.632864 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:43.632919 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.632920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.633730 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.633731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.634904 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.634907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.635622 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.635623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.635694 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.635699 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.635746 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.635785 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 11:33:43.636085 10318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a2b63c144b4e588e8a9a4195fafbeb" candidate_uuid: "a97ab72ec734489c8cff99acbc81b178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3ec40c3f3a44f958349012a3bda33e1"
I20260620 11:33:43.636162 10318 raft_consensus.cc:3037] T 77a2b63c144b4e588e8a9a4195fafbeb P c3ec40c3f3a44f958349012a3bda33e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:43.636230 10182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a2b63c144b4e588e8a9a4195fafbeb" candidate_uuid: "355a1e4b20b64b58864cc1ebd763ea12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a97ab72ec734489c8cff99acbc81b178" is_pre_election: true
I20260620 11:33:43.636332 10182 raft_consensus.cc:2379] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 355a1e4b20b64b58864cc1ebd763ea12 in current term 1: Already voted for candidate a97ab72ec734489c8cff99acbc81b178 in this term.
I20260620 11:33:43.635908 10045 raft_consensus.cc:3037] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:43.636564 10096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.632001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:46759 (local address 127.2.243.129:48343)
0620 11:33:43.632051 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:43.632061 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:43.632104 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:43.632544 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:43.632545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:43.632547 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:43.632611 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.632613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.634240 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.634241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.635318 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:43.635320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.636297 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.636298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:43.636381 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.636384 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:33:43.636418 (+ 34us) client_negotiation.cc:240] Negotiation successful
0620 11:33:43.636469 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 11:33:43.636765 10403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.632200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:48343 (local address 127.2.243.131:46759)
0620 11:33:43.632415 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.632417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.632425 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:33:43.632444 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.632444 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.632453 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.632676 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:43.632818 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.633575 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.635403 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.636207 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.636607 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.636616 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.636634 (+ 18us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.636674 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":109,"threads_started":1}
I20260620 11:33:43.636934 10317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a2b63c144b4e588e8a9a4195fafbeb" candidate_uuid: "355a1e4b20b64b58864cc1ebd763ea12" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3ec40c3f3a44f958349012a3bda33e1" is_pre_election: true
I20260620 11:33:43.637039 10318 raft_consensus.cc:2454] T 77a2b63c144b4e588e8a9a4195fafbeb P c3ec40c3f3a44f958349012a3bda33e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97ab72ec734489c8cff99acbc81b178 in term 1.
I20260620 11:33:43.637234 9981 leader_election.cc:302] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 355a1e4b20b64b58864cc1ebd763ea12; no voters: a97ab72ec734489c8cff99acbc81b178, c3ec40c3f3a44f958349012a3bda33e1
I20260620 11:33:43.637423 10045 raft_consensus.cc:2454] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97ab72ec734489c8cff99acbc81b178 in term 1.
I20260620 11:33:43.637485 10401 raft_consensus.cc:2726] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 11:33:43.637581 10116 leader_election.cc:302] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 355a1e4b20b64b58864cc1ebd763ea12, a97ab72ec734489c8cff99acbc81b178; no voters:
I20260620 11:33:43.637661 10399 raft_consensus.cc:2781] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:43.637806 10399 raft_consensus.cc:686] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 [term 1 LEADER]: Becoming Leader. State: Replica: a97ab72ec734489c8cff99acbc81b178, State: Running, Role: LEADER
I20260620 11:33:43.637892 10399 consensus_queue.cc:237] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 } } peers { permanent_uuid: "a97ab72ec734489c8cff99acbc81b178" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 41803 } } peers { permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 } }
I20260620 11:33:43.638657 9119 catalog_manager.cc:5495] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 reported cstate change: term changed from 0 to 1, leader changed from <none> to a97ab72ec734489c8cff99acbc81b178 (127.2.243.130). New cstate: current_term: 1 leader_uuid: "a97ab72ec734489c8cff99acbc81b178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97ab72ec734489c8cff99acbc81b178" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 41803 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 } health_report { overall_health: UNKNOWN } } }
I20260620 11:33:43.655078 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.652888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.2.243.130:41803)
0620 11:33:43.652941 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.652942 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.652950 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:33:43.652991 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.652992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.653015 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.653081 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:43.653198 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.653656 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.654353 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.654545 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.654607 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.654727 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:43.654827 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:43.654972 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.654991 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.655020 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 11:33:43.656845 10317 raft_consensus.cc:1261] T 77a2b63c144b4e588e8a9a4195fafbeb P c3ec40c3f3a44f958349012a3bda33e1 [term 1 FOLLOWER]: Refusing update from remote peer a97ab72ec734489c8cff99acbc81b178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:43.656868 10045 raft_consensus.cc:1261] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12 [term 1 FOLLOWER]: Refusing update from remote peer a97ab72ec734489c8cff99acbc81b178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:43.657208 10404 consensus_queue.cc:1035] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:43.657327 10399 consensus_queue.cc:1035] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:43.660322 10407 mvcc.cc:205] Tried to move back new op lower bound from 7298888596095848448 to 7298888596021276672. Current Snapshot: MvccSnapshot[applied={T|T < 7298888596095848448}]
I20260620 11:33:43.662953 10408 mvcc.cc:205] Tried to move back new op lower bound from 7298888596095848448 to 7298888596021276672. Current Snapshot: MvccSnapshot[applied={T|T < 7298888596095848448}]
I20260620 11:33:43.662945 9120 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:43.663030 9120 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:43.663069 10409 mvcc.cc:205] Tried to move back new op lower bound from 7298888596095848448 to 7298888596021276672. Current Snapshot: MvccSnapshot[applied={T|T < 7298888596095848448 or (T in {7298888596095848448})}]
I20260620 11:33:43.664321 9120 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5a8f84236d84cfb9ecad30874a8dc5a: Sending DeleteTablet for 3 replicas of tablet 77a2b63c144b4e588e8a9a4195fafbeb
I20260620 11:33:43.664637 10162 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a2b63c144b4e588e8a9a4195fafbeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:43 UTC) from {username='oryx', principal='oryx/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:60552
I20260620 11:33:43.664682 10025 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a2b63c144b4e588e8a9a4195fafbeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:43 UTC) from {username='oryx', principal='oryx/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:49908
I20260620 11:33:43.664870 10416 tablet_replica.cc:330] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178: stopping tablet replica
I20260620 11:33:43.664968 10416 raft_consensus.cc:2229] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:33:43.665060 10416 raft_consensus.cc:2258] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:43.665223 10298 tablet_service.cc:1514] Processing DeleteTablet for tablet 77a2b63c144b4e588e8a9a4195fafbeb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:43 UTC) from {username='oryx', principal='oryx/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:58452
I20260620 11:33:43.665737 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 9088
I20260620 11:33:43.667001 10418 tablet_replica.cc:330] T 77a2b63c144b4e588e8a9a4195fafbeb P c3ec40c3f3a44f958349012a3bda33e1: stopping tablet replica
I20260620 11:33:43.667111 10418 raft_consensus.cc:2229] T 77a2b63c144b4e588e8a9a4195fafbeb P c3ec40c3f3a44f958349012a3bda33e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:43.667220 10418 raft_consensus.cc:2258] T 77a2b63c144b4e588e8a9a4195fafbeb P c3ec40c3f3a44f958349012a3bda33e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:43.667661 10418 ts_tablet_manager.cc:1905] T 77a2b63c144b4e588e8a9a4195fafbeb P c3ec40c3f3a44f958349012a3bda33e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:43.669333 10418 ts_tablet_manager.cc:1918] T 77a2b63c144b4e588e8a9a4195fafbeb P c3ec40c3f3a44f958349012a3bda33e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:33:43.669421 10418 log.cc:1196] T 77a2b63c144b4e588e8a9a4195fafbeb P c3ec40c3f3a44f958349012a3bda33e1: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/77a2b63c144b4e588e8a9a4195fafbeb
I20260620 11:33:43.669770 10418 ts_tablet_manager.cc:1939] T 77a2b63c144b4e588e8a9a4195fafbeb P c3ec40c3f3a44f958349012a3bda33e1: Deleting consensus metadata
W20260620 11:33:43.673254 10254 connection.cc:535] server connection from 127.0.0.1:58452 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 11:33:43.673436 10381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.665500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52040 (local address 127.2.243.188:38025)
0620 11:33:43.665539 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.665542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.665550 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:33:43.665595 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.665596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.665607 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.665646 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.666375 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.666931 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.668383 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.668698 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.668744 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.670572 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.670573 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.670573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.670581 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.672136 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.672511 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.672512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.672512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.672572 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.672773 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.672774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.672775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.672932 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.673028 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.673254 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.673342 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:33:43.675120 10416 ts_tablet_manager.cc:1905] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:43.675426 10417 tablet_replica.cc:330] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12: stopping tablet replica
I20260620 11:33:43.675549 10417 raft_consensus.cc:2229] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:43.675805 10417 raft_consensus.cc:2258] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:43.676288 10417 ts_tablet_manager.cc:1905] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:43.677214 10416 ts_tablet_manager.cc:1918] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:33:43.677342 10416 log.cc:1196] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/wal/wals/77a2b63c144b4e588e8a9a4195fafbeb
I20260620 11:33:43.677672 10416 ts_tablet_manager.cc:1939] T 77a2b63c144b4e588e8a9a4195fafbeb P a97ab72ec734489c8cff99acbc81b178: Deleting consensus metadata
I20260620 11:33:43.678010 10417 ts_tablet_manager.cc:1918] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:33:43.678087 10417 log.cc:1196] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-0/wal/wals/77a2b63c144b4e588e8a9a4195fafbeb
W20260620 11:33:43.678097 10117 connection.cc:441] server connection from 127.0.0.1:60552 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60552 (request call id 1) could send its response
I20260620 11:33:43.678332 10417 ts_tablet_manager.cc:1939] T 77a2b63c144b4e588e8a9a4195fafbeb P 355a1e4b20b64b58864cc1ebd763ea12: Deleting consensus metadata
W20260620 11:33:43.678692 9980 connection.cc:441] server connection from 127.0.0.1:49908 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49908 (request call id 1) could send its response
I20260620 11:33:43.684660 10381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.675137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52054 (local address 127.2.243.188:38025)
0620 11:33:43.675177 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.675179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.675500 (+ 321us) server_negotiation.cc:407] Connection header received
0620 11:33:43.675592 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.675593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.675603 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.675632 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.676425 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.677192 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.678963 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.679300 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.679375 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.681651 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.681652 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.681652 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.681660 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.683407 (+ 1747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.683817 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.683818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.683819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.683876 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.684059 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.684060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.684061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.684220 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.684329 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.684440 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.684520 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:33:43.685880 10380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:43.675305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53706 (local address 127.2.243.189:38923)
0620 11:33:43.675351 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:43.675353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:43.679014 (+ 3661us) server_negotiation.cc:407] Connection header received
0620 11:33:43.679077 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:43.679078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:43.679093 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:43.679149 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:43.679802 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.680563 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.681259 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:43.681478 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:43.681564 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:43.683015 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:43.683016 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:43.683017 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:43.683024 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:43.684754 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.685152 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.685153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.685153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.685215 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:43.685401 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:43.685402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:43.685403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:43.685531 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:43.685624 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:43.685700 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 11:33:43.685787 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 11:33:44.661872 10427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:44.661618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.2.243.131:51363)
0620 11:33:44.661759 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:44.661827 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
W20260620 11:33:44.662099 10364 heartbeater.cc:644] Failed to heartbeat to 127.2.243.190:41191 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
I20260620 11:33:44.662781 10428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:44.662532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.2.243.129:52061)
0620 11:33:44.662661 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:44.662737 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":58,"threads_started":1}
W20260620 11:33:44.663064 10091 heartbeater.cc:644] Failed to heartbeat to 127.2.243.190:41191 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
I20260620 11:33:44.663795 10429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:44.663640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.2.243.130:55873)
0620 11:33:44.663720 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:44.663765 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69,"thread_start_us":40,"threads_started":1}
W20260620 11:33:44.663985 10228 heartbeater.cc:644] Failed to heartbeat to 127.2.243.190:41191 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
I20260620 11:33:45.164784 10430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a5a8f84236d84cfb9ecad30874a8dc5a)
I20260620 11:33:45.164959 10430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: false } }
I20260620 11:33:45.165237 10430 leader_election.cc:288] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5a8f84236d84cfb9ecad30874a8dc5a (127.2.243.190:41191), 46d54b8fefdc47e7b1e30ffe0d1adc23 (127.2.243.188:38025)
I20260620 11:33:45.165467 10431 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a5a8f84236d84cfb9ecad30874a8dc5a)
I20260620 11:33:45.165632 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.165401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.0.0.1:44540)
0620 11:33:45.165518 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.165587 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":49,"threads_started":1}
I20260620 11:33:45.165594 10431 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: false } }
W20260620 11:33:45.165766 9705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
I20260620 11:33:45.165864 10431 leader_election.cc:288] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a5a8f84236d84cfb9ecad30874a8dc5a (127.2.243.190:41191), 3aa17d267b434b8781609a0171af2bca (127.2.243.189:38923)
I20260620 11:33:45.166271 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.166070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.0.0.1:44544)
0620 11:33:45.166215 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.166243 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260620 11:33:45.166334 10437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.165940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.0.0.1:44542)
0620 11:33:45.166251 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.166297 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":47,"threads_started":1}
W20260620 11:33:45.166359 9705 leader_election.cc:334] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a5a8f84236d84cfb9ecad30874a8dc5a (127.2.243.190:41191): Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
W20260620 11:33:45.166424 9899 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
I20260620 11:33:45.166971 10437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.166851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.0.0.1:44560)
0620 11:33:45.166920 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.166945 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260620 11:33:45.167045 9899 leader_election.cc:334] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a5a8f84236d84cfb9ecad30874a8dc5a (127.2.243.190:41191): Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
I20260620 11:33:45.168776 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.165413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:38025 (local address 127.0.0.1:52060)
0620 11:33:45.165641 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.165654 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:45.165829 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:45.166027 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:45.166028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:45.166030 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:45.166086 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:45.166089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:45.167071 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:45.167072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:45.168002 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:45.168004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:45.168572 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:45.168573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:45.168625 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:45.168629 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:33:45.168697 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 11:33:45.168731 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":50,"threads_started":1}
I20260620 11:33:45.168818 10434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.165607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52060 (local address 127.2.243.188:38025)
0620 11:33:45.165772 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:45.165775 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:45.165786 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:33:45.165923 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:45.165924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:45.165935 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:45.166022 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:45.166151 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:45.166981 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:45.168077 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:45.168498 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:45.168563 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:45.168574 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:45.168719 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 11:33:45.168758 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":116,"threads_started":1}
I20260620 11:33:45.168988 9922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3aa17d267b434b8781609a0171af2bca" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" is_pre_election: true
I20260620 11:33:45.169085 9922 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3aa17d267b434b8781609a0171af2bca in term 1.
I20260620 11:33:45.169225 9703 leader_election.cc:302] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aa17d267b434b8781609a0171af2bca, 46d54b8fefdc47e7b1e30ffe0d1adc23; no voters: a5a8f84236d84cfb9ecad30874a8dc5a
I20260620 11:33:45.169337 10430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 11:33:45.169389 10430 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 1 FOLLOWER]: Starting leader election (detected failure of leader a5a8f84236d84cfb9ecad30874a8dc5a)
I20260620 11:33:45.169409 10436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.166084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:38923 (local address 127.0.0.1:53720)
0620 11:33:45.166464 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.166475 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:45.166541 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:45.166702 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:45.166705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:45.166707 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:45.166749 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:45.166750 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:45.167588 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:45.167589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:45.168363 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:45.168365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:45.169121 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:45.169122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:45.169204 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:45.169209 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:33:45.169262 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 11:33:45.169326 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":41,"threads_started":1}
I20260620 11:33:45.169410 10438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.166098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53720 (local address 127.2.243.189:38923)
0620 11:33:45.166248 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:45.166249 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:45.166525 (+ 276us) server_negotiation.cc:407] Connection header received
0620 11:33:45.166607 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:45.166607 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:45.166616 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:45.166688 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:45.166812 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:45.167511 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:45.168466 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:45.169022 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:45.169095 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:45.169106 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:45.169279 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 11:33:45.169315 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260620 11:33:45.169413 10430 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 1 FOLLOWER]: Advancing to term 2
I20260620 11:33:45.169636 9728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3aa17d267b434b8781609a0171af2bca" is_pre_election: true
I20260620 11:33:45.170243 10430 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: false } }
I20260620 11:33:45.170375 9728 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46d54b8fefdc47e7b1e30ffe0d1adc23 in current term 2: Already voted for candidate 3aa17d267b434b8781609a0171af2bca in this term.
I20260620 11:33:45.170392 10430 leader_election.cc:288] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [CANDIDATE]: Term 2 election: Requested vote from peers a5a8f84236d84cfb9ecad30874a8dc5a (127.2.243.190:41191), 46d54b8fefdc47e7b1e30ffe0d1adc23 (127.2.243.188:38025)
I20260620 11:33:45.170539 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.170454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.0.0.1:44564)
0620 11:33:45.170485 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.170514 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 11:33:45.170542 9899 leader_election.cc:302] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46d54b8fefdc47e7b1e30ffe0d1adc23; no voters: 3aa17d267b434b8781609a0171af2bca, a5a8f84236d84cfb9ecad30874a8dc5a
I20260620 11:33:45.170697 9922 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3aa17d267b434b8781609a0171af2bca" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23"
I20260620 11:33:45.170769 9922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [term 1 FOLLOWER]: Advancing to term 2
I20260620 11:33:45.170837 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.170743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.0.0.1:44568)
0620 11:33:45.170792 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.170816 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 11:33:45.170915 9705 leader_election.cc:334] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a5a8f84236d84cfb9ecad30874a8dc5a (127.2.243.190:41191): Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
I20260620 11:33:45.171428 9922 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3aa17d267b434b8781609a0171af2bca in term 2.
I20260620 11:33:45.171540 10431 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 11:33:45.171597 9703 leader_election.cc:302] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3aa17d267b434b8781609a0171af2bca, 46d54b8fefdc47e7b1e30ffe0d1adc23; no voters: a5a8f84236d84cfb9ecad30874a8dc5a
I20260620 11:33:45.171710 10430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 2 FOLLOWER]: Leader election won for term 2
I20260620 11:33:45.171830 10430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [term 2 LEADER]: Becoming Leader. State: Replica: 3aa17d267b434b8781609a0171af2bca, State: Running, Role: LEADER
I20260620 11:33:45.171885 10430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: false } }
I20260620 11:33:45.172222 10440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aa17d267b434b8781609a0171af2bca. Latest consensus state: current_term: 2 leader_uuid: "3aa17d267b434b8781609a0171af2bca" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: false } } }
I20260620 11:33:45.172317 10440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:45.172479 10442 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:33:45.172802 10442 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:33:45.172852 10442 catalog_manager.cc:1259] Loaded cluster ID: 0cbb43cecc8540a98dd3fee84da7a872
I20260620 11:33:45.172868 10442 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:33:45.172971 10442 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:33:45.173018 10442 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca: Loaded TSK: 0
I20260620 11:33:45.173101 10442 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:33:45.229856 9718 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:33:45.231509 9718 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:33:45.232568 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.232441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.0.0.1:44584)
0620 11:33:45.232505 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.232543 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 11:33:45.232628 9922 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [term 2 FOLLOWER]: Refusing update from remote peer 3aa17d267b434b8781609a0171af2bca: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 11:33:45.232939 10440 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [LEADER]: Connected to new peer: Peer: permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 11:33:45.233009 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.232912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.0.0.1:44598)
0620 11:33:45.232962 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.232991 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 11:33:45.233134 9705 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca -> Peer a5a8f84236d84cfb9ecad30874a8dc5a (127.2.243.190:41191): Couldn't send request to peer a5a8f84236d84cfb9ecad30874a8dc5a. Status: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 11:33:45.233541 10431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aa17d267b434b8781609a0171af2bca. Latest consensus state: current_term: 2 leader_uuid: "3aa17d267b434b8781609a0171af2bca" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: false } } }
I20260620 11:33:45.233639 10431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:33:45.234037 10443 mvcc.cc:205] Tried to move back new op lower bound from 7298888602550095872 to 7298888602304458752. Current Snapshot: MvccSnapshot[applied={T|T < 7298888602550095872}]
I20260620 11:33:45.234125 10430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "3aa17d267b434b8781609a0171af2bca" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: false } } }
I20260620 11:33:45.234228 10430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:45.234359 10431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "3aa17d267b434b8781609a0171af2bca" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a5a8f84236d84cfb9ecad30874a8dc5a" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41191 } } peers { permanent_uuid: "3aa17d267b434b8781609a0171af2bca" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 38923 } attrs { promote: false } } peers { permanent_uuid: "46d54b8fefdc47e7b1e30ffe0d1adc23" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 38025 } attrs { promote: false } } }
I20260620 11:33:45.234429 10431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d54b8fefdc47e7b1e30ffe0d1adc23 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:33:45.239871 10433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.236420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:42977 (local address 127.0.0.1:49916)
0620 11:33:45.236565 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.236575 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:45.236624 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:45.237148 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:45.237149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:45.237151 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:45.237205 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:45.237206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:45.238121 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:45.238122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:45.238916 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:45.238919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:45.239632 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:45.239633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:45.239699 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:45.239703 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:33:45.239760 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 11:33:45.239799 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129}
I20260620 11:33:45.239925 10449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.236486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49916 (local address 127.2.243.129:42977)
0620 11:33:45.236991 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:45.236993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:45.237002 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:33:45.237035 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:45.237036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:45.237048 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:45.237141 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:45.237289 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:45.237996 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:45.239012 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:45.239546 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:45.239628 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:45.239641 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:45.239778 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 11:33:45.239821 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":62,"threads_started":1}
I20260620 11:33:45.240139 10432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.236521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:41803 (local address 127.0.0.1:60570)
0620 11:33:45.236832 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.236843 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:45.236876 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:45.237094 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:45.237095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:45.237096 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:45.237169 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:45.237170 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:45.238075 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:45.238076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:45.239141 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:45.239143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:45.239922 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:45.239923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:45.239983 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:45.239989 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 11:33:45.240026 (+ 37us) client_negotiation.cc:240] Negotiation successful
0620 11:33:45.240067 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":21,"thread_start_us":97,"threads_started":1}
I20260620 11:33:45.240247 10451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.236727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58466 (local address 127.2.243.131:46759)
0620 11:33:45.236918 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:45.236920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:45.236927 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:33:45.236952 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:45.236953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:45.236963 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:45.237023 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:45.237250 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:45.237760 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:45.239376 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:45.239872 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:45.239934 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:45.239946 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:45.240116 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 11:33:45.240167 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":79,"threads_started":1}
I20260620 11:33:45.240267 10450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.236476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:46759 (local address 127.0.0.1:58466)
0620 11:33:45.236660 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.236671 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:45.236830 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:45.237079 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:45.237080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:45.237082 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:45.237169 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:45.237170 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:45.237846 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:45.237848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:45.239304 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:45.239305 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:45.239979 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:45.239980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:45.240052 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:45.240056 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:33:45.240105 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 11:33:45.240143 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":53}
I20260620 11:33:45.240440 10025 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff3e06548074a129b744946064f2168 (DEFAULT_TABLE table=test-table [id=5f88e15db7c8454c96d08b3c416c1971]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:45.240543 10298 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff3e06548074a129b744946064f2168 (DEFAULT_TABLE table=test-table [id=5f88e15db7c8454c96d08b3c416c1971]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:45.240578 10025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff3e06548074a129b744946064f2168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:45.240625 10452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.236726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.2.243.130:41803)
0620 11:33:45.236857 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:45.236859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:45.236868 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:33:45.236942 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:45.236944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:45.236956 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:45.237069 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:45.237229 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:45.238000 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:45.239222 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:45.239821 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:45.240449 (+ 628us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:45.240466 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:45.240487 (+ 21us) server_negotiation.cc:299] Negotiation successful
0620 11:33:45.240544 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260620 11:33:45.240677 10298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff3e06548074a129b744946064f2168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:45.240806 10162 tablet_service.cc:1467] Processing CreateTablet for tablet 2ff3e06548074a129b744946064f2168 (DEFAULT_TABLE table=test-table [id=5f88e15db7c8454c96d08b3c416c1971]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:45.240913 10162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ff3e06548074a129b744946064f2168. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:45.242230 10454 tablet_bootstrap.cc:492] T 2ff3e06548074a129b744946064f2168 P c3ec40c3f3a44f958349012a3bda33e1: Bootstrap starting.
I20260620 11:33:45.242442 10453 tablet_bootstrap.cc:492] T 2ff3e06548074a129b744946064f2168 P 355a1e4b20b64b58864cc1ebd763ea12: Bootstrap starting.
I20260620 11:33:45.242558 10455 tablet_bootstrap.cc:492] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178: Bootstrap starting.
I20260620 11:33:45.242643 10454 tablet_bootstrap.cc:654] T 2ff3e06548074a129b744946064f2168 P c3ec40c3f3a44f958349012a3bda33e1: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:45.242910 10453 tablet_bootstrap.cc:654] T 2ff3e06548074a129b744946064f2168 P 355a1e4b20b64b58864cc1ebd763ea12: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:45.242956 10455 tablet_bootstrap.cc:654] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:45.243363 10454 tablet_bootstrap.cc:492] T 2ff3e06548074a129b744946064f2168 P c3ec40c3f3a44f958349012a3bda33e1: No bootstrap required, opened a new log
I20260620 11:33:45.243445 10454 ts_tablet_manager.cc:1397] T 2ff3e06548074a129b744946064f2168 P c3ec40c3f3a44f958349012a3bda33e1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 11:33:45.243507 10455 tablet_bootstrap.cc:492] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178: No bootstrap required, opened a new log
I20260620 11:33:45.243541 10453 tablet_bootstrap.cc:492] T 2ff3e06548074a129b744946064f2168 P 355a1e4b20b64b58864cc1ebd763ea12: No bootstrap required, opened a new log
I20260620 11:33:45.243570 10455 ts_tablet_manager.cc:1397] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:45.243580 10453 ts_tablet_manager.cc:1397] T 2ff3e06548074a129b744946064f2168 P 355a1e4b20b64b58864cc1ebd763ea12: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:45.243579 10454 raft_consensus.cc:348] T 2ff3e06548074a129b744946064f2168 P c3ec40c3f3a44f958349012a3bda33e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 } } peers { permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 } } peers { permanent_uuid: "a97ab72ec734489c8cff99acbc81b178" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 41803 } }
I20260620 11:33:45.243688 10455 raft_consensus.cc:348] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 } } peers { permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 } } peers { permanent_uuid: "a97ab72ec734489c8cff99acbc81b178" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 41803 } }
I20260620 11:33:45.243764 10455 raft_consensus.cc:374] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:45.243719 10453 raft_consensus.cc:348] T 2ff3e06548074a129b744946064f2168 P 355a1e4b20b64b58864cc1ebd763ea12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 } } peers { permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 } } peers { permanent_uuid: "a97ab72ec734489c8cff99acbc81b178" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 41803 } }
I20260620 11:33:45.243810 10455 raft_consensus.cc:729] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a97ab72ec734489c8cff99acbc81b178, State: Initialized, Role: FOLLOWER
I20260620 11:33:45.243825 10453 raft_consensus.cc:374] T 2ff3e06548074a129b744946064f2168 P 355a1e4b20b64b58864cc1ebd763ea12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:45.243850 10453 raft_consensus.cc:729] T 2ff3e06548074a129b744946064f2168 P 355a1e4b20b64b58864cc1ebd763ea12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 355a1e4b20b64b58864cc1ebd763ea12, State: Initialized, Role: FOLLOWER
I20260620 11:33:45.243772 10454 raft_consensus.cc:374] T 2ff3e06548074a129b744946064f2168 P c3ec40c3f3a44f958349012a3bda33e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:45.243866 10455 consensus_queue.cc:260] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 } } peers { permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 } } peers { permanent_uuid: "a97ab72ec734489c8cff99acbc81b178" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 41803 } }
I20260620 11:33:45.243917 10454 raft_consensus.cc:729] T 2ff3e06548074a129b744946064f2168 P c3ec40c3f3a44f958349012a3bda33e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3ec40c3f3a44f958349012a3bda33e1, State: Initialized, Role: FOLLOWER
I20260620 11:33:45.243891 10453 consensus_queue.cc:260] T 2ff3e06548074a129b744946064f2168 P 355a1e4b20b64b58864cc1ebd763ea12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 } } peers { permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 } } peers { permanent_uuid: "a97ab72ec734489c8cff99acbc81b178" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 41803 } }
I20260620 11:33:45.243979 10454 consensus_queue.cc:260] T 2ff3e06548074a129b744946064f2168 P c3ec40c3f3a44f958349012a3bda33e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 } } peers { permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 } } peers { permanent_uuid: "a97ab72ec734489c8cff99acbc81b178" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 41803 } }
I20260620 11:33:45.244042 10453 ts_tablet_manager.cc:1428] T 2ff3e06548074a129b744946064f2168 P 355a1e4b20b64b58864cc1ebd763ea12: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:45.244037 10455 ts_tablet_manager.cc:1428] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 11:33:45.244307 10454 ts_tablet_manager.cc:1428] T 2ff3e06548074a129b744946064f2168 P c3ec40c3f3a44f958349012a3bda33e1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 11:33:45.244675 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.244320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.2.243.130:55845)
0620 11:33:45.244568 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.244611 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":80,"threads_started":1}
I20260620 11:33:45.244987 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.244730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.2.243.131:45471)
0620 11:33:45.244915 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.244949 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":72,"threads_started":1}
I20260620 11:33:45.245133 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.245025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.2.243.130:49427)
0620 11:33:45.245074 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.245105 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 11:33:45.245339 10093 heartbeater.cc:503] Master 127.2.243.189:38923 was elected leader, sending a full tablet report...
I20260620 11:33:45.245895 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.245800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.2.243.131:57697)
0620 11:33:45.245836 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.245868 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 11:33:45.246037 10366 heartbeater.cc:503] Master 127.2.243.189:38923 was elected leader, sending a full tablet report...
I20260620 11:33:45.246312 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.244238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.2.243.129:37731)
0620 11:33:45.246205 (+ 1967us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.246278 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1945,"thread_start_us":69,"threads_started":1}
I20260620 11:33:45.246687 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.246582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.2.243.129:38967)
0620 11:33:45.246637 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.246662 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 11:33:45.271987 10094 tablet.cc:2369] T 2ff3e06548074a129b744946064f2168 P 355a1e4b20b64b58864cc1ebd763ea12: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 11:33:45.297887 10367 tablet.cc:2369] T 2ff3e06548074a129b744946064f2168 P c3ec40c3f3a44f958349012a3bda33e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:33:45.309748 10459 raft_consensus.cc:482] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:33:45.309876 10459 raft_consensus.cc:504] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 } } peers { permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 } } peers { permanent_uuid: "a97ab72ec734489c8cff99acbc81b178" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 41803 } }
I20260620 11:33:45.310110 10459 leader_election.cc:288] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 355a1e4b20b64b58864cc1ebd763ea12 (127.2.243.129:42977), c3ec40c3f3a44f958349012a3bda33e1 (127.2.243.131:46759)
I20260620 11:33:45.310286 10045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff3e06548074a129b744946064f2168" candidate_uuid: "a97ab72ec734489c8cff99acbc81b178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "355a1e4b20b64b58864cc1ebd763ea12" is_pre_election: true
I20260620 11:33:45.310374 10045 raft_consensus.cc:2454] T 2ff3e06548074a129b744946064f2168 P 355a1e4b20b64b58864cc1ebd763ea12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97ab72ec734489c8cff99acbc81b178 in term 0.
I20260620 11:33:45.310335 10317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff3e06548074a129b744946064f2168" candidate_uuid: "a97ab72ec734489c8cff99acbc81b178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3ec40c3f3a44f958349012a3bda33e1" is_pre_election: true
I20260620 11:33:45.310469 10317 raft_consensus.cc:2454] T 2ff3e06548074a129b744946064f2168 P c3ec40c3f3a44f958349012a3bda33e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a97ab72ec734489c8cff99acbc81b178 in term 0.
I20260620 11:33:45.310529 10116 leader_election.cc:302] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 355a1e4b20b64b58864cc1ebd763ea12, a97ab72ec734489c8cff99acbc81b178; no voters:
I20260620 11:33:45.310645 10459 raft_consensus.cc:2781] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:33:45.310708 10459 raft_consensus.cc:482] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:33:45.310724 10459 raft_consensus.cc:3037] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:45.311437 10459 raft_consensus.cc:504] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 } } peers { permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 } } peers { permanent_uuid: "a97ab72ec734489c8cff99acbc81b178" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 41803 } }
I20260620 11:33:45.311621 10459 leader_election.cc:288] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178 [CANDIDATE]: Term 1 election: Requested vote from peers 355a1e4b20b64b58864cc1ebd763ea12 (127.2.243.129:42977), c3ec40c3f3a44f958349012a3bda33e1 (127.2.243.131:46759)
I20260620 11:33:45.311725 10045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff3e06548074a129b744946064f2168" candidate_uuid: "a97ab72ec734489c8cff99acbc81b178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "355a1e4b20b64b58864cc1ebd763ea12"
I20260620 11:33:45.311807 10045 raft_consensus.cc:3037] T 2ff3e06548074a129b744946064f2168 P 355a1e4b20b64b58864cc1ebd763ea12 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:45.311829 10317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2ff3e06548074a129b744946064f2168" candidate_uuid: "a97ab72ec734489c8cff99acbc81b178" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3ec40c3f3a44f958349012a3bda33e1"
I20260620 11:33:45.311879 10317 raft_consensus.cc:3037] T 2ff3e06548074a129b744946064f2168 P c3ec40c3f3a44f958349012a3bda33e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:45.312443 10317 raft_consensus.cc:2454] T 2ff3e06548074a129b744946064f2168 P c3ec40c3f3a44f958349012a3bda33e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97ab72ec734489c8cff99acbc81b178 in term 1.
I20260620 11:33:45.312453 10045 raft_consensus.cc:2454] T 2ff3e06548074a129b744946064f2168 P 355a1e4b20b64b58864cc1ebd763ea12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a97ab72ec734489c8cff99acbc81b178 in term 1.
I20260620 11:33:45.312613 10118 leader_election.cc:302] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a97ab72ec734489c8cff99acbc81b178, c3ec40c3f3a44f958349012a3bda33e1; no voters:
I20260620 11:33:45.312721 10459 raft_consensus.cc:2781] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:45.312911 10459 raft_consensus.cc:686] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178 [term 1 LEADER]: Becoming Leader. State: Replica: a97ab72ec734489c8cff99acbc81b178, State: Running, Role: LEADER
I20260620 11:33:45.312960 10459 consensus_queue.cc:237] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 } } peers { permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 } } peers { permanent_uuid: "a97ab72ec734489c8cff99acbc81b178" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 41803 } }
I20260620 11:33:45.313102 10230 heartbeater.cc:503] Master 127.2.243.189:38923 was elected leader, sending a full tablet report...
I20260620 11:33:45.313529 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.313388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.2.243.130:45601)
0620 11:33:45.313463 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.313492 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 11:33:45.313913 9717 catalog_manager.cc:5495] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178 reported cstate change: term changed from 0 to 1, leader changed from <none> to a97ab72ec734489c8cff99acbc81b178 (127.2.243.130). New cstate: current_term: 1 leader_uuid: "a97ab72ec734489c8cff99acbc81b178" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a97ab72ec734489c8cff99acbc81b178" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 41803 } health_report { overall_health: HEALTHY } } }
I20260620 11:33:45.331497 10452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.329253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60572 (local address 127.2.243.130:41803)
0620 11:33:45.329299 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:45.329301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:45.329312 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:33:45.329377 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:45.329378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:45.329388 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:45.329437 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:45.329551 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:45.330030 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:45.330778 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:45.330996 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:45.331078 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:45.331236 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:45.331274 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:45.331333 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:45.331394 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 11:33:45.331415 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:33:45.332427 10317 raft_consensus.cc:1261] T 2ff3e06548074a129b744946064f2168 P c3ec40c3f3a44f958349012a3bda33e1 [term 1 FOLLOWER]: Refusing update from remote peer a97ab72ec734489c8cff99acbc81b178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:45.332481 10045 raft_consensus.cc:1261] T 2ff3e06548074a129b744946064f2168 P 355a1e4b20b64b58864cc1ebd763ea12 [term 1 FOLLOWER]: Refusing update from remote peer a97ab72ec734489c8cff99acbc81b178: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:45.332690 10465 consensus_queue.cc:1035] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3ec40c3f3a44f958349012a3bda33e1" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 46759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:45.332800 10459 consensus_queue.cc:1035] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178 [LEADER]: Connected to new peer: Peer: permanent_uuid: "355a1e4b20b64b58864cc1ebd763ea12" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 42977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:45.333362 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.333251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.2.243.129:57393)
0620 11:33:45.333293 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.333327 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 11:33:45.333596 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.333503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.2.243.131:56583)
0620 11:33:45.333545 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.333568 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 11:33:45.334512 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.334408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.2.243.131:53827)
0620 11:33:45.334461 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.334486 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 11:33:45.334627 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.334484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.2.243.130:52685)
0620 11:33:45.334575 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.334601 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260620 11:33:45.335373 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.335265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:41191 (local address 127.2.243.129:47907)
0620 11:33:45.335311 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.335342 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 11:33:45.336248 9718 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:45.336340 9718 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:45.337474 9718 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3aa17d267b434b8781609a0171af2bca: Sending DeleteTablet for 3 replicas of tablet 2ff3e06548074a129b744946064f2168
I20260620 11:33:45.337785 10298 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ff3e06548074a129b744946064f2168 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:45 UTC) from {username='oryx', principal='oryx/127.2.243.189@KRBTEST.COM'} at 127.0.0.1:58466
I20260620 11:33:45.337805 10162 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ff3e06548074a129b744946064f2168 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:45 UTC) from {username='oryx', principal='oryx/127.2.243.189@KRBTEST.COM'} at 127.0.0.1:60570
I20260620 11:33:45.337823 10025 tablet_service.cc:1514] Processing DeleteTablet for tablet 2ff3e06548074a129b744946064f2168 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:45 UTC) from {username='oryx', principal='oryx/127.2.243.189@KRBTEST.COM'} at 127.0.0.1:49916
I20260620 11:33:45.338044 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 9865
I20260620 11:33:45.338382 10477 tablet_replica.cc:330] T 2ff3e06548074a129b744946064f2168 P c3ec40c3f3a44f958349012a3bda33e1: stopping tablet replica
I20260620 11:33:45.338442 10477 raft_consensus.cc:2229] T 2ff3e06548074a129b744946064f2168 P c3ec40c3f3a44f958349012a3bda33e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:45.338485 10477 raft_consensus.cc:2258] T 2ff3e06548074a129b744946064f2168 P c3ec40c3f3a44f958349012a3bda33e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:45.338568 10479 tablet_replica.cc:330] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178: stopping tablet replica
I20260620 11:33:45.338807 10477 ts_tablet_manager.cc:1905] T 2ff3e06548074a129b744946064f2168 P c3ec40c3f3a44f958349012a3bda33e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:45.340689 10477 ts_tablet_manager.cc:1918] T 2ff3e06548074a129b744946064f2168 P c3ec40c3f3a44f958349012a3bda33e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:33:45.340760 10477 log.cc:1196] T 2ff3e06548074a129b744946064f2168 P c3ec40c3f3a44f958349012a3bda33e1: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/2ff3e06548074a129b744946064f2168
I20260620 11:33:45.340976 10477 ts_tablet_manager.cc:1939] T 2ff3e06548074a129b744946064f2168 P c3ec40c3f3a44f958349012a3bda33e1: Deleting consensus metadata
I20260620 11:33:45.341130 10479 raft_consensus.cc:2229] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:33:45.341223 10479 raft_consensus.cc:2258] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:45.341495 10479 ts_tablet_manager.cc:1905] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:45.342242 9705 catalog_manager.cc:4841] TS c3ec40c3f3a44f958349012a3bda33e1 (127.2.243.131:46759): tablet 2ff3e06548074a129b744946064f2168 (table test-table [id=5f88e15db7c8454c96d08b3c416c1971]) successfully deleted
I20260620 11:33:45.343330 10479 ts_tablet_manager.cc:1918] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:33:45.343391 10479 log.cc:1196] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955192827595-3022-0/minicluster-data/ts-1/wal/wals/2ff3e06548074a129b744946064f2168
I20260620 11:33:45.343612 10479 ts_tablet_manager.cc:1939] T 2ff3e06548074a129b744946064f2168 P a97ab72ec734489c8cff99acbc81b178: Deleting consensus metadata
I20260620 11:33:45.344036 9703 catalog_manager.cc:4841] TS a97ab72ec734489c8cff99acbc81b178 (127.2.243.130:41803): tablet 2ff3e06548074a129b744946064f2168 (table test-table [id=5f88e15db7c8454c96d08b3c416c1971]) successfully deleted
I20260620 11:33:45.346329 10450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:45.346218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:42977 (local address 127.0.0.1:49926)
0620 11:33:45.346270 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:45.346302 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.129:42977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 11:33:45.346359 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 10099
W20260620 11:33:45.346493 9703 catalog_manager.cc:4584] TS 355a1e4b20b64b58864cc1ebd763ea12 (127.2.243.129:42977): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2ff3e06548074a129b744946064f2168: Network error: Client connection negotiation failed: client connection to 127.2.243.129:42977: connect: Connection refused (error 111)
I20260620 11:33:45.352717 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 10235
I20260620 11:33:45.357343 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 9584
I20260620 11:33:45.361897 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 9772
2026-06-20T11:33:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4588 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:33:45 dist-test-slave-ctbw krb5kdc[10482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:33:45 dist-test-slave-ctbw krb5kdc[10482](info): set up 2 sockets
Jun 20 11:33:45 dist-test-slave-ctbw krb5kdc[10482](info): commencing operation
krb5kdc: starting...
W20260620 11:33:46.463624 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:33:46 dist-test-slave-ctbw krb5kdc[10482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:33:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:33:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.190.keytab.
Entry for principal oryx/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.190.keytab.
I20260620 11:33:46.581394 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:33311
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.190.keytab
--principal=oryx/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:33311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:46.661361 10495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:46.661549 10495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:46.661571 10495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:46.662717 10495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:33:46.662760 10495 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 11:33:46.662774 10495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:46.662787 10495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:46.662799 10495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:33:46.662812 10495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:33:46.664326 10495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:33311
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.190.keytab
--principal=oryx/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:33311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:46.664562 10495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:46.664798 10495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:46.669509 10501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:46.669513 10503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:46.669513 10500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:46.669896 10495 server_base.cc:1034] running on GCE node
I20260620 11:33:46.670116 10495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:46.670357 10495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:46.671525 10495 hybrid_clock.cc:648] HybridClock initialized: now 1781955226671516 us; error 36 us; skew 500 ppm
Jun 20 11:33:46 dist-test-slave-ctbw krb5kdc[10482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955226, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:46.673997 10495 init.cc:377] Logged in from keytab as oryx/127.2.243.190@KRBTEST.COM (short username oryx)
I20260620 11:33:46.674422 10495 webserver.cc:460] Webserver started at http://127.2.243.190:35973/ using document root <none> and password file <none>
I20260620 11:33:46.674656 10495 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:46.674726 10495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:46.674919 10495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:46.676002 10495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "676204a6b302407493c222f7219cd526"
format_stamp: "Formatted at 2026-06-20 11:33:46 on dist-test-slave-ctbw"
server_key: "2ad551d9dd2b096e6fae62e0c3de6e1d"
server_key_iv: "c769200c42e1b52eb05c3c59870f6d3f"
server_key_version: "encryptionkey@0"
I20260620 11:33:46.676360 10495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "676204a6b302407493c222f7219cd526"
format_stamp: "Formatted at 2026-06-20 11:33:46 on dist-test-slave-ctbw"
server_key: "2ad551d9dd2b096e6fae62e0c3de6e1d"
server_key_iv: "c769200c42e1b52eb05c3c59870f6d3f"
server_key_version: "encryptionkey@0"
I20260620 11:33:46.677627 10495 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 11:33:46.678359 10510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:46.678520 10495 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:33:46.678608 10495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "676204a6b302407493c222f7219cd526"
format_stamp: "Formatted at 2026-06-20 11:33:46 on dist-test-slave-ctbw"
server_key: "2ad551d9dd2b096e6fae62e0c3de6e1d"
server_key_iv: "c769200c42e1b52eb05c3c59870f6d3f"
server_key_version: "encryptionkey@0"
I20260620 11:33:46.678687 10495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:46.695197 10495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:46.697580 10495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:46.697741 10495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:46.702183 10495 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:33311
I20260620 11:33:46.702255 10572 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:33311 every 8 connection(s)
I20260620 11:33:46.702610 10495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:33:46.703358 10573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:46.705552 10573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526: Bootstrap starting.
I20260620 11:33:46.706001 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 10495
I20260620 11:33:46.706115 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:33:46.706210 10573 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:46.706349 3022 external_mini_cluster.cc:1428] Setting key 00ff7bf3f7012344458448cae9f44437
I20260620 11:33:46.706569 10573 log.cc:824] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:46.707363 10573 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526: No bootstrap required, opened a new log
I20260620 11:33:46.709229 10573 raft_consensus.cc:348] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676204a6b302407493c222f7219cd526" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33311 } }
I20260620 11:33:46.709414 10573 raft_consensus.cc:374] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:46.709450 10573 raft_consensus.cc:729] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 676204a6b302407493c222f7219cd526, State: Initialized, Role: FOLLOWER
I20260620 11:33:46.709570 10573 consensus_queue.cc:260] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676204a6b302407493c222f7219cd526" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33311 } }
I20260620 11:33:46.709640 10573 raft_consensus.cc:388] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:33:46.709668 10573 raft_consensus.cc:482] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:33:46.709714 10573 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:46.710361 10573 raft_consensus.cc:504] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676204a6b302407493c222f7219cd526" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33311 } }
I20260620 11:33:46.710449 10573 leader_election.cc:302] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 676204a6b302407493c222f7219cd526; no voters:
I20260620 11:33:46.710637 10573 leader_election.cc:288] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:33:46.710667 10578 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:46.710775 10578 raft_consensus.cc:686] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526 [term 1 LEADER]: Becoming Leader. State: Replica: 676204a6b302407493c222f7219cd526, State: Running, Role: LEADER
I20260620 11:33:46.710904 10573 sys_catalog.cc:564] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:46.710858 10578 consensus_queue.cc:237] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676204a6b302407493c222f7219cd526" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33311 } }
I20260620 11:33:46.711205 10578 sys_catalog.cc:455] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 676204a6b302407493c222f7219cd526. Latest consensus state: current_term: 1 leader_uuid: "676204a6b302407493c222f7219cd526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676204a6b302407493c222f7219cd526" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33311 } } }
I20260620 11:33:46.711285 10578 sys_catalog.cc:458] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526 [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:46.711215 10579 sys_catalog.cc:455] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "676204a6b302407493c222f7219cd526" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "676204a6b302407493c222f7219cd526" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33311 } } }
I20260620 11:33:46.711624 10579 sys_catalog.cc:458] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526 [sys.catalog]: This master's current role is: LEADER
W20260620 11:33:46.712953 10593 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:33:46.713012 10593 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:33:46.713073 10594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:33:46.713238 10594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 11:33:46 dist-test-slave-ctbw krb5kdc[10482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM
I20260620 11:33:46.714954 10594 catalog_manager.cc:1347] Generated new cluster ID: 5fe9c37b7f624b3f9362174179e013f6
I20260620 11:33:46.715008 10594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:33:46.718444 10577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:46.706876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50062 (local address 127.2.243.190:33311)
0620 11:33:46.707050 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:46.707059 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:46.707092 (+ 33us) server_negotiation.cc:407] Connection header received
0620 11:33:46.707651 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:46.707660 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:46.707721 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:46.707886 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:46.708472 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:46.710190 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:46.711297 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:46.711948 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:46.712055 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:46.715174 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:46.715178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:46.715178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:46.715191 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:46.716913 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:46.717284 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:46.717287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:46.717287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:46.717385 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:46.717601 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:46.717602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:46.717602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:46.717829 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:46.717994 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:46.718110 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 11:33:46.718245 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
I20260620 11:33:46.721987 10594 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:33:46.722627 10594 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:33:46.725386 10594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526: Generated new TSK 0
I20260620 11:33:46.725561 10594 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.2.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.129.keytab.
Entry for principal oryx/127.2.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.129.keytab.
I20260620 11:33:46.757517 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33311
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.129.keytab
--principal=oryx/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:46.858181 10599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:46.858376 10599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:46.858398 10599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:46.859552 10599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:46.859593 10599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:46.859614 10599 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 11:33:46.859632 10599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:33:46.861111 10599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.129.keytab
--principal=oryx/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.243.190:33311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:46.861348 10599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:46.861582 10599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:46.867640 10604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:46.867645 10605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:46.867645 10607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:46.867908 10599 server_base.cc:1034] running on GCE node
I20260620 11:33:46.868088 10599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:46.868328 10599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:46.869488 10599 hybrid_clock.cc:648] HybridClock initialized: now 1781955226869475 us; error 31 us; skew 500 ppm
Jun 20 11:33:46 dist-test-slave-ctbw krb5kdc[10482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955226, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:46.872089 10599 init.cc:377] Logged in from keytab as oryx/127.2.243.129@KRBTEST.COM (short username oryx)
I20260620 11:33:46.872490 10599 webserver.cc:460] Webserver started at http://127.2.243.129:40155/ using document root <none> and password file <none>
I20260620 11:33:46.872785 10599 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:46.872853 10599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:46.872985 10599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:46.874013 10599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "64dc8cdf08c5445d9ca51a87379c7df1"
format_stamp: "Formatted at 2026-06-20 11:33:46 on dist-test-slave-ctbw"
server_key: "2eaf3e75d5c19642f3e04d1a3c268a88"
server_key_iv: "0e64f26c6dc3eca9d317ab203096b62f"
server_key_version: "encryptionkey@0"
I20260620 11:33:46.874372 10599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "64dc8cdf08c5445d9ca51a87379c7df1"
format_stamp: "Formatted at 2026-06-20 11:33:46 on dist-test-slave-ctbw"
server_key: "2eaf3e75d5c19642f3e04d1a3c268a88"
server_key_iv: "0e64f26c6dc3eca9d317ab203096b62f"
server_key_version: "encryptionkey@0"
I20260620 11:33:46.875836 10599 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:33:46.876595 10614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:46.876772 10599 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:46.876871 10599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "64dc8cdf08c5445d9ca51a87379c7df1"
format_stamp: "Formatted at 2026-06-20 11:33:46 on dist-test-slave-ctbw"
server_key: "2eaf3e75d5c19642f3e04d1a3c268a88"
server_key_iv: "0e64f26c6dc3eca9d317ab203096b62f"
server_key_version: "encryptionkey@0"
I20260620 11:33:46.876943 10599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:46.891185 10599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:46.893127 10599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:46.893277 10599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:46.893754 10599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:46.893815 10599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:46.893860 10599 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:46.893886 10599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:46.900317 10599 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:36721
I20260620 11:33:46.900419 10729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:36721 every 8 connection(s)
I20260620 11:33:46.900731 10599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
Jun 20 11:33:46 dist-test-slave-ctbw krb5kdc[10482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955226, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM
I20260620 11:33:46.903126 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 10599
I20260620 11:33:46.903251 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:33:46.903478 3022 external_mini_cluster.cc:1428] Setting key 0485145fffebbc68d9ca6730160ca0a2
I20260620 11:33:46.904738 10623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:46.894247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33311 (local address 127.2.243.129:45659)
0620 11:33:46.894411 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:46.894449 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:46.895036 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:46.895383 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:46.895388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:46.895788 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:46.896072 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:46.896079 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:46.897171 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:46.897173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:46.898538 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:46.898541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:46.899128 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:46.899129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:46.899283 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:46.899820 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:46.899829 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:46.901559 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:46.903207 (+ 1648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:46.903214 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:46.903217 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:46.903524 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:46.903758 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:46.903759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:46.903760 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:46.903905 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:46.904285 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:46.904289 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:46.904396 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 11:33:46.904480 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 11:33:46.904593 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
I20260620 11:33:46.904834 10577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:46.894284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:45659 (local address 127.2.243.190:33311)
0620 11:33:46.894357 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:46.894360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:46.894537 (+ 177us) server_negotiation.cc:407] Connection header received
0620 11:33:46.895163 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:46.895165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:46.895180 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:46.895267 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:46.896185 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:46.897056 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:46.898631 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:46.899024 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:46.899116 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:46.901642 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:46.901643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:46.901644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:46.901650 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:46.903128 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:46.903625 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:46.903626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:46.903627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:46.903686 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:46.904008 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:46.904009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:46.904009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:46.904201 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:46.904287 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:46.904531 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 11:33:46.904624 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 11:33:46.905892 10730 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33311
I20260620 11:33:46.905995 10730 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:46.906196 10730 heartbeater.cc:511] Master 127.2.243.190:33311 requested a full tablet report, sending...
I20260620 11:33:46.906816 10526 ts_manager.cc:194] Registered new tserver with Master: 64dc8cdf08c5445d9ca51a87379c7df1 (127.2.243.129:36721)
I20260620 11:33:46.907764 10526 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:45659
WARNING: no policy specified for oryx/127.2.243.130@KRBTEST.COM; defaulting to no policy
I20260620 11:33:46.913024 10737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:46.906441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33311 (local address 127.2.243.129:60923)
0620 11:33:46.906740 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:46.906752 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:46.906857 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:46.907057 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:46.907058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:46.907216 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:46.907270 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:46.907271 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:46.908007 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:46.908007 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:46.908804 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:46.908806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:46.909410 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:46.909411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:46.909467 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:46.909925 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:46.909929 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:46.910235 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:46.912045 (+ 1810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:46.912046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:46.912046 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:46.912203 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:46.912418 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:46.912418 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:46.912419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:46.912466 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:46.912738 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:46.912739 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:46.912781 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 11:33:46.912837 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 11:33:46.912955 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":64,"threads_started":1}
I20260620 11:33:46.913059 10577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:46.906492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:60923 (local address 127.2.243.190:33311)
0620 11:33:46.906536 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:46.906538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:46.906830 (+ 292us) server_negotiation.cc:407] Connection header received
0620 11:33:46.906951 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:46.906952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:46.906963 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:46.906995 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:46.907365 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:46.907888 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:46.908888 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:46.909232 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:46.909326 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:46.910331 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:46.910332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:46.910333 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:46.910340 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:46.911964 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:46.912288 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:46.912290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:46.912290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:46.912347 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:46.912525 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:46.912526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:46.912527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:46.912643 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:46.912706 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:46.912892 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 11:33:46.912969 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.130.keytab.
Entry for principal oryx/127.2.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.130.keytab.
I20260620 11:33:46.929260 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33311
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.130.keytab
--principal=oryx/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:47.009044 10739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:47.009236 10739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:47.009259 10739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:47.010393 10739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:47.010432 10739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:47.010474 10739 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 11:33:47.010517 10739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:33:47.012102 10739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.130.keytab
--principal=oryx/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.243.190:33311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10739
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:47.012274 10739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:47.012447 10739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:47.017267 10745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:47.017277 10747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:47.017267 10744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:47.017450 10739 server_base.cc:1034] running on GCE node
I20260620 11:33:47.017825 10739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:47.018105 10739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:47.019261 10739 hybrid_clock.cc:648] HybridClock initialized: now 1781955227019248 us; error 30 us; skew 500 ppm
Jun 20 11:33:47 dist-test-slave-ctbw krb5kdc[10482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955227, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:47.021592 10739 init.cc:377] Logged in from keytab as oryx/127.2.243.130@KRBTEST.COM (short username oryx)
I20260620 11:33:47.021994 10739 webserver.cc:460] Webserver started at http://127.2.243.130:44777/ using document root <none> and password file <none>
I20260620 11:33:47.022213 10739 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:47.022281 10739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:47.022435 10739 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:47.023581 10739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "8f1ca528ad364d9983417d8b1e442ae5"
format_stamp: "Formatted at 2026-06-20 11:33:47 on dist-test-slave-ctbw"
server_key: "d8f2f56c2877d940d3cd02ba9a4e2033"
server_key_iv: "f0763a5a57dbfec59aabf2426fb24dc2"
server_key_version: "encryptionkey@0"
I20260620 11:33:47.023938 10739 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "8f1ca528ad364d9983417d8b1e442ae5"
format_stamp: "Formatted at 2026-06-20 11:33:47 on dist-test-slave-ctbw"
server_key: "d8f2f56c2877d940d3cd02ba9a4e2033"
server_key_iv: "f0763a5a57dbfec59aabf2426fb24dc2"
server_key_version: "encryptionkey@0"
I20260620 11:33:47.025265 10739 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:33:47.026117 10754 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:47.026335 10739 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:47.026454 10739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "8f1ca528ad364d9983417d8b1e442ae5"
format_stamp: "Formatted at 2026-06-20 11:33:47 on dist-test-slave-ctbw"
server_key: "d8f2f56c2877d940d3cd02ba9a4e2033"
server_key_iv: "f0763a5a57dbfec59aabf2426fb24dc2"
server_key_version: "encryptionkey@0"
I20260620 11:33:47.026527 10739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:47.041551 10739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:47.043948 10739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:47.044142 10739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:47.044746 10739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:47.044824 10739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:47.044869 10739 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:47.044929 10739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 11:33:47 dist-test-slave-ctbw krb5kdc[10482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955227, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM
I20260620 11:33:47.053161 10739 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:33403
I20260620 11:33:47.053228 10869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:33403 every 8 connection(s)
I20260620 11:33:47.053570 10739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:33:47.053740 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 10739
I20260620 11:33:47.054162 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:33:47.054423 3022 external_mini_cluster.cc:1428] Setting key f2d8df46025df36af9e72890b0640a19
I20260620 11:33:47.056169 10577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.045325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:33329 (local address 127.2.243.190:33311)
0620 11:33:47.045417 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:47.045421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:47.045566 (+ 145us) server_negotiation.cc:407] Connection header received
0620 11:33:47.046049 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:47.046050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:47.046071 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:47.046174 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:47.047038 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.047975 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.049033 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.049234 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.049326 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.052993 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:47.052995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:47.052995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:47.053003 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:47.054674 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:47.055193 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:47.055194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:47.055195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:47.055257 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:47.055531 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:47.055532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:47.055533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:47.055652 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:47.055727 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:47.056000 (+ 273us) server_negotiation.cc:299] Negotiation successful
0620 11:33:47.056079 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 11:33:47.056348 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.045245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33311 (local address 127.2.243.130:33329)
0620 11:33:47.045426 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:47.045466 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:47.045933 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:47.046249 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:47.046252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:47.046594 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:47.046915 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.046922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.048087 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.048088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.048936 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.048938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.049334 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.049335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.049465 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.050051 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:47.050063 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:47.052888 (+ 2825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:47.054767 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:47.054780 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:47.054782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:47.055123 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:47.055331 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:47.055332 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:47.055332 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:47.055447 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:47.055719 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:47.055724 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:47.055867 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 11:33:47.055968 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 11:33:47.056120 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":102,"threads_started":1}
I20260620 11:33:47.056861 10870 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33311
I20260620 11:33:47.056967 10870 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:47.057184 10870 heartbeater.cc:511] Master 127.2.243.190:33311 requested a full tablet report, sending...
I20260620 11:33:47.057600 10526 ts_manager.cc:194] Registered new tserver with Master: 8f1ca528ad364d9983417d8b1e442ae5 (127.2.243.130:33403)
I20260620 11:33:47.058106 10526 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:33329
WARNING: no policy specified for oryx/127.2.243.131@KRBTEST.COM; defaulting to no policy
I20260620 11:33:47.064430 10577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.057965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:41925 (local address 127.2.243.190:33311)
0620 11:33:47.057999 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:47.058001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:47.059061 (+ 1060us) server_negotiation.cc:407] Connection header received
0620 11:33:47.059133 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:47.059134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:47.059143 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:47.059198 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:47.059506 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.059969 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.060666 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.060873 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.060958 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.061890 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:47.061892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:47.061892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:47.061899 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:47.063285 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:47.063648 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:47.063649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:47.063650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:47.063707 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:47.063913 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:47.063914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:47.063914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:47.064052 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:47.064130 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:47.064267 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 11:33:47.064337 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 11:33:47.064435 10877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.057917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33311 (local address 127.2.243.130:41925)
0620 11:33:47.058993 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:47.059003 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:47.059074 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:47.059203 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:47.059204 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:47.059380 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:47.059426 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.059427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.060062 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.060063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.060570 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.060572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.060959 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.060960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.061025 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.061473 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:47.061477 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:47.061737 (+ 260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:47.063393 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:47.063394 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:47.063395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:47.063564 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:47.063771 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:47.063772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:47.063772 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:47.063849 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:47.064122 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:47.064122 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:47.064167 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 11:33:47.064241 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 11:33:47.064339 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1056,"thread_start_us":73,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.131.keytab.
Entry for principal oryx/127.2.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.131.keytab.
I20260620 11:33:47.081000 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33311
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.131.keytab
--principal=oryx/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:46869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:47.161932 10879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:47.162140 10879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:47.162164 10879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:47.163338 10879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:47.163379 10879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:47.163399 10879 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 11:33:47.163417 10879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:33:47.164924 10879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:46869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.131.keytab
--principal=oryx/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.2.243.190:33311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:47.165177 10879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:47.165417 10879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:47.170352 10887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:47.170362 10884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:47.170352 10885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:47.170933 10879 server_base.cc:1034] running on GCE node
I20260620 11:33:47.171147 10879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:47.171389 10879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:47.172556 10879 hybrid_clock.cc:648] HybridClock initialized: now 1781955227172551 us; error 45 us; skew 500 ppm
Jun 20 11:33:47 dist-test-slave-ctbw krb5kdc[10482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955227, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:47.175130 10879 init.cc:377] Logged in from keytab as oryx/127.2.243.131@KRBTEST.COM (short username oryx)
I20260620 11:33:47.175648 10879 webserver.cc:460] Webserver started at http://127.2.243.131:44497/ using document root <none> and password file <none>
I20260620 11:33:47.175886 10879 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:47.175954 10879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:47.176090 10879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:47.177146 10879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "ae7b39323eec4a86bc8f8f27cb547a1d"
format_stamp: "Formatted at 2026-06-20 11:33:47 on dist-test-slave-ctbw"
server_key: "6420cfcc473eea752ee3d0262d50749e"
server_key_iv: "b03d7f71149188b0bba3748836e8e333"
server_key_version: "encryptionkey@0"
I20260620 11:33:47.177528 10879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "ae7b39323eec4a86bc8f8f27cb547a1d"
format_stamp: "Formatted at 2026-06-20 11:33:47 on dist-test-slave-ctbw"
server_key: "6420cfcc473eea752ee3d0262d50749e"
server_key_iv: "b03d7f71149188b0bba3748836e8e333"
server_key_version: "encryptionkey@0"
I20260620 11:33:47.178990 10879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:33:47.179813 10894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:47.180019 10879 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:47.180120 10879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "ae7b39323eec4a86bc8f8f27cb547a1d"
format_stamp: "Formatted at 2026-06-20 11:33:47 on dist-test-slave-ctbw"
server_key: "6420cfcc473eea752ee3d0262d50749e"
server_key_iv: "b03d7f71149188b0bba3748836e8e333"
server_key_version: "encryptionkey@0"
I20260620 11:33:47.180209 10879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:47.191905 10879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:47.194193 10879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:47.194353 10879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:47.194938 10879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:47.194993 10879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:47.195027 10879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:47.195055 10879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 11:33:47 dist-test-slave-ctbw krb5kdc[10482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955227, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM
I20260620 11:33:47.202291 10879 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:43581
I20260620 11:33:47.202406 11009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:43581 every 8 connection(s)
I20260620 11:33:47.202713 10879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:33:47.204288 10577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.195316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:43233 (local address 127.2.243.190:33311)
0620 11:33:47.195383 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:47.195385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:47.195634 (+ 249us) server_negotiation.cc:407] Connection header received
0620 11:33:47.196116 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:47.196118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:47.196131 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:47.196211 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:47.196778 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.197597 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.198610 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.198825 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.198925 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.201402 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:47.201404 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:47.201404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:47.201412 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:47.203234 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:47.203580 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:47.203581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:47.203581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:47.203632 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:47.203828 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:47.203829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:47.203829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:47.203925 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:47.203973 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:47.204171 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 11:33:47.204227 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 11:33:47.204485 10903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.195320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33311 (local address 127.2.243.131:43233)
0620 11:33:47.195478 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:47.195547 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:47.196019 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:47.196261 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:47.196264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:47.196495 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:47.196679 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.196683 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.197675 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.197676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.198516 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.198518 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.198931 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.198932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.199091 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.199565 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:47.199573 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:47.201305 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:47.203303 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:47.203307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:47.203309 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:47.203512 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:47.203689 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:47.203690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:47.203690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:47.203776 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:47.203977 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:47.203981 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:47.204082 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 11:33:47.204160 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 11:33:47.204270 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260620 11:33:47.204998 11010 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33311
I20260620 11:33:47.205072 11010 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:47.205256 11010 heartbeater.cc:511] Master 127.2.243.190:33311 requested a full tablet report, sending...
I20260620 11:33:47.205657 10527 ts_manager.cc:194] Registered new tserver with Master: ae7b39323eec4a86bc8f8f27cb547a1d (127.2.243.131:43581)
I20260620 11:33:47.205814 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 10879
I20260620 11:33:47.205947 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:33:47.206164 10527 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:43233
I20260620 11:33:47.206187 3022 external_mini_cluster.cc:1428] Setting key 4e0ae5e66d14c05f04c9fa0c077a5eb4
I20260620 11:33:47.207044 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:33:47.212981 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.206049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33311 (local address 127.2.243.131:40643)
0620 11:33:47.206281 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:47.206293 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:47.206355 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:47.207036 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:47.207037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:47.207224 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:47.207279 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.207280 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.207949 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.207950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.208824 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.208827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.209302 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.209302 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.209347 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.209769 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:47.209773 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:47.210034 (+ 261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:47.211923 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:47.211924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:47.211924 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:47.212114 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:47.212341 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:47.212342 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:47.212343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:47.212417 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:47.212717 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:47.212718 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:47.212770 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 11:33:47.212844 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 11:33:47.212917 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":140,"threads_started":1}
I20260620 11:33:47.213104 10577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.206092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:40643 (local address 127.2.243.190:33311)
0620 11:33:47.206148 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:47.206150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:47.206342 (+ 192us) server_negotiation.cc:407] Connection header received
0620 11:33:47.206412 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:47.206413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:47.206423 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:47.206470 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:47.207399 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.207846 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.208921 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.209225 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.209299 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.210121 (+ 822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:47.210122 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:47.210123 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:47.210130 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:47.211803 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:47.212187 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:47.212188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:47.212189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:47.212246 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:47.212495 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:47.212496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:47.212496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:47.212635 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:47.212713 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:47.212886 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 11:33:47.212966 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Jun 20 11:33:47 dist-test-slave-ctbw krb5kdc[10482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955227, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 11:33:47 dist-test-slave-ctbw krb5kdc[10482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:47 dist-test-slave-ctbw krb5kdc[10482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 11:33:47.229645 10577 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:47.223793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50064 (local address 127.2.243.190:33311)
0620 11:33:47.223924 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:47.223927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:47.224041 (+ 114us) server_negotiation.cc:407] Connection header received
0620 11:33:47.224105 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:47.224106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:47.224116 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:47.224205 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:47.224763 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.225335 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.226205 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.226401 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.226473 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.229533 (+ 3060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50064: BlockingRecv error: recv got EOF from 127.0.0.1:50064 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Jun 20 11:33:47 dist-test-slave-ctbw krb5kdc[10482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955227, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM
I20260620 11:33:47.236783 10577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.230472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50076 (local address 127.2.243.190:33311)
0620 11:33:47.230521 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:47.230523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:47.230632 (+ 109us) server_negotiation.cc:407] Connection header received
0620 11:33:47.230661 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:47.230662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:47.230670 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:47.230718 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:47.231227 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.231674 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.232310 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.232474 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.232532 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.234686 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:47.234687 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:47.234687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:47.234693 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:47.235911 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:47.236214 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:47.236214 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:47.236215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:47.236260 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:47.236421 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:47.236421 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:47.236422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:47.236509 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:47.236590 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:47.236669 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 11:33:47.236721 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 11:33:47.238510 10527 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:33:47.240136 10527 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:33:47.246650 11033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.242562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:36721 (local address 127.0.0.1:51256)
0620 11:33:47.242829 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:47.242839 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:47.243274 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:47.243548 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:47.243549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:47.243551 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:47.243587 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.243588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.244744 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.244745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.245630 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.245632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.246317 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.246318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.246397 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.246411 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 11:33:47.246512 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 11:33:47.246566 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":87,"threads_started":1}
I20260620 11:33:47.246665 11034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.242660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51256 (local address 127.2.243.129:36721)
0620 11:33:47.243140 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:47.243145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:47.243395 (+ 250us) server_negotiation.cc:407] Connection header received
0620 11:33:47.243435 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:47.243438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:47.243455 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:47.243551 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:47.243648 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.244666 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.245714 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.246215 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.246315 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.246394 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:47.246535 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 11:33:47.246572 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":69,"threads_started":1}
I20260620 11:33:47.246835 11032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.242499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:43581 (local address 127.0.0.1:60602)
0620 11:33:47.242717 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:47.242762 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:47.242830 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:47.243213 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:47.243216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:47.243218 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:47.243282 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.243284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.244583 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.244584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.245669 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.245671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.246580 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.246582 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.246657 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.246660 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:33:47.246689 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 11:33:47.246733 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":8,"thread_start_us":69,"threads_started":1}
I20260620 11:33:47.246836 11036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.242722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:33403 (local address 127.0.0.1:35682)
0620 11:33:47.242958 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:47.242969 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:47.243023 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:47.243310 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:47.243310 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:47.243313 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:47.243365 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.243367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.244320 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.244321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.245671 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.245673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.246582 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.246583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.246651 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.246655 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:33:47.246687 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 11:33:47.246733 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":15,"thread_start_us":63,"threads_started":1}
I20260620 11:33:47.247257 11037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.242953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35682 (local address 127.2.243.130:33403)
0620 11:33:47.243149 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:47.243153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:47.243161 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:33:47.243195 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:47.243197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:47.243210 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:47.243434 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:47.243494 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.244234 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.245768 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.246476 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.246901 (+ 425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.247036 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:47.247102 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 11:33:47.247151 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":104,"threads_started":1}
I20260620 11:33:47.247673 10664 tablet_service.cc:1467] Processing CreateTablet for tablet 2a71d4c516f44188b9b44ba901855a25 (DEFAULT_TABLE table=test-table [id=1408176282f144c58c41ac2e1ece53af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:47.248013 10664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a71d4c516f44188b9b44ba901855a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:47.247903 10804 tablet_service.cc:1467] Processing CreateTablet for tablet 2a71d4c516f44188b9b44ba901855a25 (DEFAULT_TABLE table=test-table [id=1408176282f144c58c41ac2e1ece53af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:47.248260 10804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a71d4c516f44188b9b44ba901855a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:47.251071 11035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.242678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60602 (local address 127.2.243.131:43581)
0620 11:33:47.243027 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:47.243031 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:47.243038 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:33:47.243096 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:47.243099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:47.243115 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:47.243207 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:47.243371 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.244497 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.245768 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.246484 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.246959 (+ 475us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.247207 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:47.247236 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 11:33:47.247279 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":68,"threads_started":1}
I20260620 11:33:47.251297 11039 tablet_bootstrap.cc:492] T 2a71d4c516f44188b9b44ba901855a25 P 8f1ca528ad364d9983417d8b1e442ae5: Bootstrap starting.
I20260620 11:33:47.251559 11038 tablet_bootstrap.cc:492] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1: Bootstrap starting.
I20260620 11:33:47.251747 10944 tablet_service.cc:1467] Processing CreateTablet for tablet 2a71d4c516f44188b9b44ba901855a25 (DEFAULT_TABLE table=test-table [id=1408176282f144c58c41ac2e1ece53af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:47.251904 11039 tablet_bootstrap.cc:654] T 2a71d4c516f44188b9b44ba901855a25 P 8f1ca528ad364d9983417d8b1e442ae5: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:47.252110 10944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a71d4c516f44188b9b44ba901855a25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:47.252192 11038 tablet_bootstrap.cc:654] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:47.252264 11039 log.cc:824] T 2a71d4c516f44188b9b44ba901855a25 P 8f1ca528ad364d9983417d8b1e442ae5: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:47.252488 11038 log.cc:824] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:47.252982 11039 tablet_bootstrap.cc:492] T 2a71d4c516f44188b9b44ba901855a25 P 8f1ca528ad364d9983417d8b1e442ae5: No bootstrap required, opened a new log
I20260620 11:33:47.253064 11039 ts_tablet_manager.cc:1397] T 2a71d4c516f44188b9b44ba901855a25 P 8f1ca528ad364d9983417d8b1e442ae5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:33:47.254554 11039 raft_consensus.cc:348] T 2a71d4c516f44188b9b44ba901855a25 P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } }
I20260620 11:33:47.254703 11039 raft_consensus.cc:374] T 2a71d4c516f44188b9b44ba901855a25 P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:47.254748 11039 raft_consensus.cc:729] T 2a71d4c516f44188b9b44ba901855a25 P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1ca528ad364d9983417d8b1e442ae5, State: Initialized, Role: FOLLOWER
I20260620 11:33:47.254884 11039 consensus_queue.cc:260] T 2a71d4c516f44188b9b44ba901855a25 P 8f1ca528ad364d9983417d8b1e442ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } }
I20260620 11:33:47.255074 11042 tablet_bootstrap.cc:492] T 2a71d4c516f44188b9b44ba901855a25 P ae7b39323eec4a86bc8f8f27cb547a1d: Bootstrap starting.
I20260620 11:33:47.255160 11039 ts_tablet_manager.cc:1428] T 2a71d4c516f44188b9b44ba901855a25 P 8f1ca528ad364d9983417d8b1e442ae5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:33:47.255316 10870 heartbeater.cc:503] Master 127.2.243.190:33311 was elected leader, sending a full tablet report...
I20260620 11:33:47.255792 11038 tablet_bootstrap.cc:492] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1: No bootstrap required, opened a new log
I20260620 11:33:47.255846 11038 ts_tablet_manager.cc:1397] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 11:33:47.257362 11038 raft_consensus.cc:348] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } }
I20260620 11:33:47.257484 11038 raft_consensus.cc:374] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:47.257505 11038 raft_consensus.cc:729] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64dc8cdf08c5445d9ca51a87379c7df1, State: Initialized, Role: FOLLOWER
I20260620 11:33:47.257644 11038 consensus_queue.cc:260] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } }
I20260620 11:33:47.257911 11038 ts_tablet_manager.cc:1428] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:33:47.257918 10730 heartbeater.cc:503] Master 127.2.243.190:33311 was elected leader, sending a full tablet report...
I20260620 11:33:47.258214 11042 tablet_bootstrap.cc:654] T 2a71d4c516f44188b9b44ba901855a25 P ae7b39323eec4a86bc8f8f27cb547a1d: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:47.258565 11042 log.cc:824] T 2a71d4c516f44188b9b44ba901855a25 P ae7b39323eec4a86bc8f8f27cb547a1d: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:47.259315 11042 tablet_bootstrap.cc:492] T 2a71d4c516f44188b9b44ba901855a25 P ae7b39323eec4a86bc8f8f27cb547a1d: No bootstrap required, opened a new log
I20260620 11:33:47.259368 11042 ts_tablet_manager.cc:1397] T 2a71d4c516f44188b9b44ba901855a25 P ae7b39323eec4a86bc8f8f27cb547a1d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 11:33:47.260666 11042 raft_consensus.cc:348] T 2a71d4c516f44188b9b44ba901855a25 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } }
I20260620 11:33:47.260782 11042 raft_consensus.cc:374] T 2a71d4c516f44188b9b44ba901855a25 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:47.260803 11042 raft_consensus.cc:729] T 2a71d4c516f44188b9b44ba901855a25 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae7b39323eec4a86bc8f8f27cb547a1d, State: Initialized, Role: FOLLOWER
I20260620 11:33:47.260878 11042 consensus_queue.cc:260] T 2a71d4c516f44188b9b44ba901855a25 P ae7b39323eec4a86bc8f8f27cb547a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } }
I20260620 11:33:47.261121 11042 ts_tablet_manager.cc:1428] T 2a71d4c516f44188b9b44ba901855a25 P ae7b39323eec4a86bc8f8f27cb547a1d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:33:47.261152 11010 heartbeater.cc:503] Master 127.2.243.190:33311 was elected leader, sending a full tablet report...
I20260620 11:33:47.264091 11044 raft_consensus.cc:482] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:33:47.264159 11044 raft_consensus.cc:504] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } }
I20260620 11:33:47.264375 11044 leader_election.cc:288] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae7b39323eec4a86bc8f8f27cb547a1d (127.2.243.131:43581), 8f1ca528ad364d9983417d8b1e442ae5 (127.2.243.130:33403)
I20260620 11:33:47.267489 11037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.264778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:33587 (local address 127.2.243.130:33403)
0620 11:33:47.264819 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:47.264820 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:47.264942 (+ 122us) server_negotiation.cc:407] Connection header received
0620 11:33:47.264973 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:47.264974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:47.264983 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:47.265085 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:47.265216 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.265687 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.266601 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.267151 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.267217 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.267223 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:47.267387 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 11:33:47.267424 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:33:47.267489 11035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.264645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:38333 (local address 127.2.243.131:43581)
0620 11:33:47.264697 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:47.264699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:47.264713 (+ 14us) server_negotiation.cc:407] Connection header received
0620 11:33:47.264742 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:47.264743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:47.264753 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:47.264814 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:47.265040 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.265570 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.266547 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.267113 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.267191 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.267202 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:47.267387 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 11:33:47.267424 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 11:33:47.267555 10623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.264578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:43581 (local address 127.2.243.129:38333)
0620 11:33:47.264634 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:47.264646 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:47.264687 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:47.264870 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:47.264872 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:47.264873 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:47.264957 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.264959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.265641 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.265641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.266447 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.266449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.267202 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.267203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.267300 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.267304 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:33:47.267370 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 11:33:47.267408 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":5}
I20260620 11:33:47.267743 11047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.264696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:33403 (local address 127.2.243.129:33587)
0620 11:33:47.264812 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:47.264820 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:47.264904 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:47.265092 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:47.265092 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:47.265094 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:47.265153 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.265154 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.265768 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.265768 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.266530 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.266533 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.267220 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.267221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.267300 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.267311 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 11:33:47.267370 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 11:33:47.267408 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":81,"threads_started":1}
I20260620 11:33:47.267901 10963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a71d4c516f44188b9b44ba901855a25" candidate_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" is_pre_election: true
I20260620 11:33:47.268064 10963 raft_consensus.cc:2454] T 2a71d4c516f44188b9b44ba901855a25 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64dc8cdf08c5445d9ca51a87379c7df1 in term 0.
I20260620 11:33:47.268126 10824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a71d4c516f44188b9b44ba901855a25" candidate_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1ca528ad364d9983417d8b1e442ae5" is_pre_election: true
I20260620 11:33:47.268277 10824 raft_consensus.cc:2454] T 2a71d4c516f44188b9b44ba901855a25 P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64dc8cdf08c5445d9ca51a87379c7df1 in term 0.
I20260620 11:33:47.268270 10618 leader_election.cc:302] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64dc8cdf08c5445d9ca51a87379c7df1, ae7b39323eec4a86bc8f8f27cb547a1d; no voters:
I20260620 11:33:47.268419 11044 raft_consensus.cc:2781] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:33:47.268528 11044 raft_consensus.cc:482] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:33:47.268568 11044 raft_consensus.cc:3037] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:47.269397 11044 raft_consensus.cc:504] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } }
I20260620 11:33:47.269522 11044 leader_election.cc:288] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1 [CANDIDATE]: Term 1 election: Requested vote from peers ae7b39323eec4a86bc8f8f27cb547a1d (127.2.243.131:43581), 8f1ca528ad364d9983417d8b1e442ae5 (127.2.243.130:33403)
I20260620 11:33:47.269691 10963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a71d4c516f44188b9b44ba901855a25" candidate_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d"
I20260620 11:33:47.269716 10824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a71d4c516f44188b9b44ba901855a25" candidate_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1ca528ad364d9983417d8b1e442ae5"
I20260620 11:33:47.269766 10963 raft_consensus.cc:3037] T 2a71d4c516f44188b9b44ba901855a25 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:47.269816 10824 raft_consensus.cc:3037] T 2a71d4c516f44188b9b44ba901855a25 P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:47.270450 10963 raft_consensus.cc:2454] T 2a71d4c516f44188b9b44ba901855a25 P ae7b39323eec4a86bc8f8f27cb547a1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64dc8cdf08c5445d9ca51a87379c7df1 in term 1.
I20260620 11:33:47.270514 10824 raft_consensus.cc:2454] T 2a71d4c516f44188b9b44ba901855a25 P 8f1ca528ad364d9983417d8b1e442ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64dc8cdf08c5445d9ca51a87379c7df1 in term 1.
I20260620 11:33:47.270668 10618 leader_election.cc:302] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64dc8cdf08c5445d9ca51a87379c7df1, 8f1ca528ad364d9983417d8b1e442ae5; no voters:
I20260620 11:33:47.270773 11044 raft_consensus.cc:2781] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:47.270946 11044 raft_consensus.cc:686] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 1 LEADER]: Becoming Leader. State: Replica: 64dc8cdf08c5445d9ca51a87379c7df1, State: Running, Role: LEADER
I20260620 11:33:47.271052 11044 consensus_queue.cc:237] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } }
I20260620 11:33:47.271755 10526 catalog_manager.cc:5495] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64dc8cdf08c5445d9ca51a87379c7df1 (127.2.243.129). New cstate: current_term: 1 leader_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } health_report { overall_health: UNKNOWN } } }
I20260620 11:33:47.281889 11034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.279716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51268 (local address 127.2.243.129:36721)
0620 11:33:47.279769 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:47.279770 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:47.279777 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:33:47.279803 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:47.279804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:47.279818 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:47.279887 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:47.280028 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.280545 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.281223 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.281408 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.281460 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.281621 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:47.281731 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:47.281807 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:47.281823 (+ 16us) server_negotiation.cc:299] Negotiation successful
0620 11:33:47.281840 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 11:33:47.283618 10824 raft_consensus.cc:1261] T 2a71d4c516f44188b9b44ba901855a25 P 8f1ca528ad364d9983417d8b1e442ae5 [term 1 FOLLOWER]: Refusing update from remote peer 64dc8cdf08c5445d9ca51a87379c7df1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:47.283668 10963 raft_consensus.cc:1261] T 2a71d4c516f44188b9b44ba901855a25 P ae7b39323eec4a86bc8f8f27cb547a1d [term 1 FOLLOWER]: Refusing update from remote peer 64dc8cdf08c5445d9ca51a87379c7df1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:47.284031 11044 consensus_queue.cc:1035] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:47.284159 11048 consensus_queue.cc:1035] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:47.286401 11051 mvcc.cc:205] Tried to move back new op lower bound from 7298888610951012352 to 7298888610902564864. Current Snapshot: MvccSnapshot[applied={T|T < 7298888610951012352}]
I20260620 11:33:47.286619 11053 mvcc.cc:205] Tried to move back new op lower bound from 7298888610951012352 to 7298888610902564864. Current Snapshot: MvccSnapshot[applied={T|T < 7298888610951012352}]
I20260620 11:33:47.287076 11052 mvcc.cc:205] Tried to move back new op lower bound from 7298888610951012352 to 7298888610902564864. Current Snapshot: MvccSnapshot[applied={T|T < 7298888610951012352}]
I20260620 11:33:47.289450 10526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:47.289548 10526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:47.290335 10526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526: Sending DeleteTablet for 3 replicas of tablet 2a71d4c516f44188b9b44ba901855a25
I20260620 11:33:47.290669 10664 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a71d4c516f44188b9b44ba901855a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:47 UTC) from {username='oryx', principal='oryx/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:51256
I20260620 11:33:47.290750 10944 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a71d4c516f44188b9b44ba901855a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:47 UTC) from {username='oryx', principal='oryx/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:60602
I20260620 11:33:47.290825 10526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 11:33:47.290966 10804 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a71d4c516f44188b9b44ba901855a25 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:47 UTC) from {username='oryx', principal='oryx/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:35682
W20260620 11:33:47.291074 10526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:33:47.291290 11060 tablet_replica.cc:330] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1: stopping tablet replica
I20260620 11:33:47.291415 11060 raft_consensus.cc:2229] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:33:47.291536 11060 raft_consensus.cc:2258] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:47.292320 11060 ts_tablet_manager.cc:1905] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:47.292321 10804 tablet_service.cc:1467] Processing CreateTablet for tablet 0268244f10d842e48e2d8dda45b28b66 (DEFAULT_TABLE table=test-table [id=68cd03faa28f47d4af27175a18987b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:47.292451 10804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0268244f10d842e48e2d8dda45b28b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:47.294168 10944 tablet_service.cc:1467] Processing CreateTablet for tablet 0268244f10d842e48e2d8dda45b28b66 (DEFAULT_TABLE table=test-table [id=68cd03faa28f47d4af27175a18987b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:47.294245 11060 ts_tablet_manager.cc:1918] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:33:47.294286 10944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0268244f10d842e48e2d8dda45b28b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:47.294312 11060 log.cc:1196] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-0/wal/wals/2a71d4c516f44188b9b44ba901855a25
I20260620 11:33:47.294348 11039 tablet_bootstrap.cc:492] T 0268244f10d842e48e2d8dda45b28b66 P 8f1ca528ad364d9983417d8b1e442ae5: Bootstrap starting.
I20260620 11:33:47.294625 11060 ts_tablet_manager.cc:1939] T 2a71d4c516f44188b9b44ba901855a25 P 64dc8cdf08c5445d9ca51a87379c7df1: Deleting consensus metadata
I20260620 11:33:47.294850 11039 tablet_bootstrap.cc:654] T 0268244f10d842e48e2d8dda45b28b66 P 8f1ca528ad364d9983417d8b1e442ae5: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:47.295789 11042 tablet_bootstrap.cc:492] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d: Bootstrap starting.
I20260620 11:33:47.295909 11039 tablet_bootstrap.cc:492] T 0268244f10d842e48e2d8dda45b28b66 P 8f1ca528ad364d9983417d8b1e442ae5: No bootstrap required, opened a new log
I20260620 11:33:47.295960 11039 ts_tablet_manager.cc:1397] T 0268244f10d842e48e2d8dda45b28b66 P 8f1ca528ad364d9983417d8b1e442ae5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 11:33:47.296120 11039 raft_consensus.cc:348] T 0268244f10d842e48e2d8dda45b28b66 P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:47.296195 11039 raft_consensus.cc:374] T 0268244f10d842e48e2d8dda45b28b66 P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:47.296197 11042 tablet_bootstrap.cc:654] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:47.296223 11039 raft_consensus.cc:729] T 0268244f10d842e48e2d8dda45b28b66 P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1ca528ad364d9983417d8b1e442ae5, State: Initialized, Role: FOLLOWER
I20260620 11:33:47.296267 11039 consensus_queue.cc:260] T 0268244f10d842e48e2d8dda45b28b66 P 8f1ca528ad364d9983417d8b1e442ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:47.296370 11039 ts_tablet_manager.cc:1428] T 0268244f10d842e48e2d8dda45b28b66 P 8f1ca528ad364d9983417d8b1e442ae5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 11:33:47.296720 10512 catalog_manager.cc:4841] TS 64dc8cdf08c5445d9ca51a87379c7df1 (127.2.243.129:36721): tablet 2a71d4c516f44188b9b44ba901855a25 (table test-table [id=1408176282f144c58c41ac2e1ece53af]) successfully deleted
I20260620 11:33:47.296806 11042 tablet_bootstrap.cc:492] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d: No bootstrap required, opened a new log
I20260620 11:33:47.296847 11042 ts_tablet_manager.cc:1397] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:47.296909 10664 tablet_service.cc:1467] Processing CreateTablet for tablet 0268244f10d842e48e2d8dda45b28b66 (DEFAULT_TABLE table=test-table [id=68cd03faa28f47d4af27175a18987b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:47.297008 10664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0268244f10d842e48e2d8dda45b28b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:47.296995 11042 raft_consensus.cc:348] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:47.297070 11042 raft_consensus.cc:374] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:47.297092 11042 raft_consensus.cc:729] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae7b39323eec4a86bc8f8f27cb547a1d, State: Initialized, Role: FOLLOWER
I20260620 11:33:47.297130 11042 consensus_queue.cc:260] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:47.297224 11042 ts_tablet_manager.cc:1428] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:47.298172 11038 tablet_bootstrap.cc:492] T 0268244f10d842e48e2d8dda45b28b66 P 64dc8cdf08c5445d9ca51a87379c7df1: Bootstrap starting.
I20260620 11:33:47.298573 11038 tablet_bootstrap.cc:654] T 0268244f10d842e48e2d8dda45b28b66 P 64dc8cdf08c5445d9ca51a87379c7df1: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:47.298978 11061 tablet_replica.cc:330] T 2a71d4c516f44188b9b44ba901855a25 P ae7b39323eec4a86bc8f8f27cb547a1d: stopping tablet replica
I20260620 11:33:47.299000 11062 tablet_replica.cc:330] T 2a71d4c516f44188b9b44ba901855a25 P 8f1ca528ad364d9983417d8b1e442ae5: stopping tablet replica
I20260620 11:33:47.299113 11061 raft_consensus.cc:2229] T 2a71d4c516f44188b9b44ba901855a25 P ae7b39323eec4a86bc8f8f27cb547a1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:47.299114 11062 raft_consensus.cc:2229] T 2a71d4c516f44188b9b44ba901855a25 P 8f1ca528ad364d9983417d8b1e442ae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:47.299225 11038 tablet_bootstrap.cc:492] T 0268244f10d842e48e2d8dda45b28b66 P 64dc8cdf08c5445d9ca51a87379c7df1: No bootstrap required, opened a new log
I20260620 11:33:47.299232 11061 raft_consensus.cc:2258] T 2a71d4c516f44188b9b44ba901855a25 P ae7b39323eec4a86bc8f8f27cb547a1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:47.299232 11062 raft_consensus.cc:2258] T 2a71d4c516f44188b9b44ba901855a25 P 8f1ca528ad364d9983417d8b1e442ae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:47.299276 11038 ts_tablet_manager.cc:1397] T 0268244f10d842e48e2d8dda45b28b66 P 64dc8cdf08c5445d9ca51a87379c7df1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:47.299415 11038 raft_consensus.cc:348] T 0268244f10d842e48e2d8dda45b28b66 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:47.299518 11038 raft_consensus.cc:374] T 0268244f10d842e48e2d8dda45b28b66 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:47.299549 11038 raft_consensus.cc:729] T 0268244f10d842e48e2d8dda45b28b66 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64dc8cdf08c5445d9ca51a87379c7df1, State: Initialized, Role: FOLLOWER
I20260620 11:33:47.299592 11038 consensus_queue.cc:260] T 0268244f10d842e48e2d8dda45b28b66 P 64dc8cdf08c5445d9ca51a87379c7df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:47.299681 11038 ts_tablet_manager.cc:1428] T 0268244f10d842e48e2d8dda45b28b66 P 64dc8cdf08c5445d9ca51a87379c7df1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:47.299711 11061 ts_tablet_manager.cc:1905] T 2a71d4c516f44188b9b44ba901855a25 P ae7b39323eec4a86bc8f8f27cb547a1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:47.300158 11062 ts_tablet_manager.cc:1905] T 2a71d4c516f44188b9b44ba901855a25 P 8f1ca528ad364d9983417d8b1e442ae5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:47.301220 11061 ts_tablet_manager.cc:1918] T 2a71d4c516f44188b9b44ba901855a25 P ae7b39323eec4a86bc8f8f27cb547a1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:33:47.301293 11061 log.cc:1196] T 2a71d4c516f44188b9b44ba901855a25 P ae7b39323eec4a86bc8f8f27cb547a1d: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/2a71d4c516f44188b9b44ba901855a25
I20260620 11:33:47.301545 11062 ts_tablet_manager.cc:1918] T 2a71d4c516f44188b9b44ba901855a25 P 8f1ca528ad364d9983417d8b1e442ae5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:33:47.301551 11061 ts_tablet_manager.cc:1939] T 2a71d4c516f44188b9b44ba901855a25 P ae7b39323eec4a86bc8f8f27cb547a1d: Deleting consensus metadata
I20260620 11:33:47.301633 11062 log.cc:1196] T 2a71d4c516f44188b9b44ba901855a25 P 8f1ca528ad364d9983417d8b1e442ae5: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-1/wal/wals/2a71d4c516f44188b9b44ba901855a25
I20260620 11:33:47.301878 11062 ts_tablet_manager.cc:1939] T 2a71d4c516f44188b9b44ba901855a25 P 8f1ca528ad364d9983417d8b1e442ae5: Deleting consensus metadata
I20260620 11:33:47.302006 10514 catalog_manager.cc:4841] TS ae7b39323eec4a86bc8f8f27cb547a1d (127.2.243.131:43581): tablet 2a71d4c516f44188b9b44ba901855a25 (table test-table [id=1408176282f144c58c41ac2e1ece53af]) successfully deleted
I20260620 11:33:47.302304 10514 catalog_manager.cc:4841] TS 8f1ca528ad364d9983417d8b1e442ae5 (127.2.243.130:33403): tablet 2a71d4c516f44188b9b44ba901855a25 (table test-table [id=1408176282f144c58c41ac2e1ece53af]) successfully deleted
W20260620 11:33:47.304847 10871 tablet.cc:2369] T 0268244f10d842e48e2d8dda45b28b66 P 8f1ca528ad364d9983417d8b1e442ae5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 11:33:47.401470 10731 tablet.cc:2369] T 0268244f10d842e48e2d8dda45b28b66 P 64dc8cdf08c5445d9ca51a87379c7df1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 11:33:47.453341 11011 tablet.cc:2369] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:33:47.688417 11046 raft_consensus.cc:482] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:33:47.688516 11046 raft_consensus.cc:504] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:47.688833 11046 leader_election.cc:288] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64dc8cdf08c5445d9ca51a87379c7df1 (127.2.243.129:36721), 8f1ca528ad364d9983417d8b1e442ae5 (127.2.243.130:33403)
I20260620 11:33:47.692133 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.689019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:33403 (local address 127.2.243.131:58157)
0620 11:33:47.689169 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:47.689186 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:47.689241 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:47.689420 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:47.689421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:47.689423 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:47.689565 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.689566 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.690273 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.690274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.691340 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.691342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.691921 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.691922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.691974 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.691978 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:33:47.692021 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 11:33:47.692053 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":106,"thread_start_us":58,"threads_started":1}
I20260620 11:33:47.692162 11037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.689100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:58157 (local address 127.2.243.130:33403)
0620 11:33:47.689144 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:47.689146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:47.689260 (+ 114us) server_negotiation.cc:407] Connection header received
0620 11:33:47.689289 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:47.689290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:47.689299 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:47.689362 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:47.689667 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.690191 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.691428 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.691815 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.691920 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.691930 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:47.692045 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 11:33:47.692092 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:33:47.692371 10824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0268244f10d842e48e2d8dda45b28b66" candidate_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1ca528ad364d9983417d8b1e442ae5" is_pre_election: true
I20260620 11:33:47.692466 10824 raft_consensus.cc:2454] T 0268244f10d842e48e2d8dda45b28b66 P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae7b39323eec4a86bc8f8f27cb547a1d in term 0.
I20260620 11:33:47.692644 10898 leader_election.cc:302] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1ca528ad364d9983417d8b1e442ae5, ae7b39323eec4a86bc8f8f27cb547a1d; no voters:
I20260620 11:33:47.692747 10903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.689015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:36721 (local address 127.2.243.131:47585)
0620 11:33:47.689064 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:47.689078 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:47.689130 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:47.689419 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:47.689421 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:47.689423 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:47.689560 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.689562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.690803 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.690804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.691706 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.691708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.692501 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.692503 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.692581 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.692585 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:33:47.692647 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 11:33:47.692689 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":6}
I20260620 11:33:47.692772 11034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.689055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:47585 (local address 127.2.243.129:36721)
0620 11:33:47.689103 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:47.689105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:47.689117 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:33:47.689253 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:47.689254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:47.689270 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:47.689331 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:47.690287 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.690728 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.691786 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.692405 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.692485 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.692494 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:47.692659 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 11:33:47.692695 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:33:47.692924 11046 raft_consensus.cc:2781] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:33:47.692965 11046 raft_consensus.cc:482] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:33:47.692984 11046 raft_consensus.cc:3037] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:47.693007 10682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0268244f10d842e48e2d8dda45b28b66" candidate_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" is_pre_election: true
I20260620 11:33:47.693164 10682 raft_consensus.cc:2454] T 0268244f10d842e48e2d8dda45b28b66 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae7b39323eec4a86bc8f8f27cb547a1d in term 0.
I20260620 11:33:47.693775 11046 raft_consensus.cc:504] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:47.693936 11046 leader_election.cc:288] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d [CANDIDATE]: Term 1 election: Requested vote from peers 64dc8cdf08c5445d9ca51a87379c7df1 (127.2.243.129:36721), 8f1ca528ad364d9983417d8b1e442ae5 (127.2.243.130:33403)
I20260620 11:33:47.694103 10682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0268244f10d842e48e2d8dda45b28b66" candidate_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64dc8cdf08c5445d9ca51a87379c7df1"
I20260620 11:33:47.694103 10824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0268244f10d842e48e2d8dda45b28b66" candidate_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1ca528ad364d9983417d8b1e442ae5"
I20260620 11:33:47.694175 10824 raft_consensus.cc:3037] T 0268244f10d842e48e2d8dda45b28b66 P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:47.694175 10682 raft_consensus.cc:3037] T 0268244f10d842e48e2d8dda45b28b66 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:47.694929 10824 raft_consensus.cc:2454] T 0268244f10d842e48e2d8dda45b28b66 P 8f1ca528ad364d9983417d8b1e442ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae7b39323eec4a86bc8f8f27cb547a1d in term 1.
I20260620 11:33:47.695005 10682 raft_consensus.cc:2454] T 0268244f10d842e48e2d8dda45b28b66 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae7b39323eec4a86bc8f8f27cb547a1d in term 1.
I20260620 11:33:47.695113 10898 leader_election.cc:302] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1ca528ad364d9983417d8b1e442ae5, ae7b39323eec4a86bc8f8f27cb547a1d; no voters:
I20260620 11:33:47.695237 11046 raft_consensus.cc:2781] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:47.695355 11046 raft_consensus.cc:686] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d [term 1 LEADER]: Becoming Leader. State: Replica: ae7b39323eec4a86bc8f8f27cb547a1d, State: Running, Role: LEADER
I20260620 11:33:47.695446 11046 consensus_queue.cc:237] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:47.696182 10526 catalog_manager.cc:5495] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d reported cstate change: term changed from 0 to 1, leader changed from <none> to ae7b39323eec4a86bc8f8f27cb547a1d (127.2.243.131). New cstate: current_term: 1 leader_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } health_report { overall_health: HEALTHY } } }
Jun 20 11:33:47 dist-test-slave-ctbw krb5kdc[10482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955226, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.190@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM
I20260620 11:33:47.763098 11074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.755766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33311 (local address 127.0.0.1:50080)
0620 11:33:47.755969 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:47.755982 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:47.756086 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:47.756240 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:47.756241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:47.756654 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:47.756717 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.756720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.757412 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.757412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.757966 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:47.757968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.758360 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.758361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:47.758425 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.759003 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:47.759008 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:47.760405 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:47.762056 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:47.762063 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:47.762065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:47.762283 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:47.762505 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:47.762506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:47.762506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:47.762568 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:47.762788 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:47.762793 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:47.762836 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 11:33:47.762915 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 11:33:47.763010 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":84,"threads_started":1}
I20260620 11:33:47.763181 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:47.755845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50080 (local address 127.2.243.190:33311)
0620 11:33:47.755972 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:47.755974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:47.756065 (+ 91us) server_negotiation.cc:407] Connection header received
0620 11:33:47.756151 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:47.756153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:47.756168 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:47.756236 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:47.756805 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.757312 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.758067 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:47.758261 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:47.758349 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:47.760492 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:47.760493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:47.760494 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:47.760500 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:47.761927 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:47.762372 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:47.762373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:47.762374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:47.762430 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:47.762626 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:47.762627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:47.762627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:47.762730 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:47.762786 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:47.762989 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 11:33:47.763058 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":45,"threads_started":1}
I20260620 11:33:47.765237 10526 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:50080:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 11:33:47.765421 10526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:33:47.766938 10664 tablet_service.cc:1467] Processing CreateTablet for tablet af48781f875d4f1898eccdfb757ecddb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6cd205896dcf4b568be52a48859d5fff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 11:33:47.766981 10944 tablet_service.cc:1467] Processing CreateTablet for tablet af48781f875d4f1898eccdfb757ecddb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6cd205896dcf4b568be52a48859d5fff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 11:33:47.766991 10804 tablet_service.cc:1467] Processing CreateTablet for tablet af48781f875d4f1898eccdfb757ecddb (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6cd205896dcf4b568be52a48859d5fff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 11:33:47.767129 10944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af48781f875d4f1898eccdfb757ecddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:47.767129 10804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af48781f875d4f1898eccdfb757ecddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:47.767153 10664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af48781f875d4f1898eccdfb757ecddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:47.768586 11038 tablet_bootstrap.cc:492] T af48781f875d4f1898eccdfb757ecddb P 64dc8cdf08c5445d9ca51a87379c7df1: Bootstrap starting.
I20260620 11:33:47.768792 11042 tablet_bootstrap.cc:492] T af48781f875d4f1898eccdfb757ecddb P ae7b39323eec4a86bc8f8f27cb547a1d: Bootstrap starting.
I20260620 11:33:47.768978 11038 tablet_bootstrap.cc:654] T af48781f875d4f1898eccdfb757ecddb P 64dc8cdf08c5445d9ca51a87379c7df1: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:47.769181 11042 tablet_bootstrap.cc:654] T af48781f875d4f1898eccdfb757ecddb P ae7b39323eec4a86bc8f8f27cb547a1d: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:47.769543 11039 tablet_bootstrap.cc:492] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5: Bootstrap starting.
I20260620 11:33:47.769589 11038 tablet_bootstrap.cc:492] T af48781f875d4f1898eccdfb757ecddb P 64dc8cdf08c5445d9ca51a87379c7df1: No bootstrap required, opened a new log
I20260620 11:33:47.769670 11038 ts_tablet_manager.cc:1397] T af48781f875d4f1898eccdfb757ecddb P 64dc8cdf08c5445d9ca51a87379c7df1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:47.769775 11038 raft_consensus.cc:348] T af48781f875d4f1898eccdfb757ecddb P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } }
I20260620 11:33:47.769867 11038 raft_consensus.cc:374] T af48781f875d4f1898eccdfb757ecddb P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:47.769903 11038 raft_consensus.cc:729] T af48781f875d4f1898eccdfb757ecddb P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64dc8cdf08c5445d9ca51a87379c7df1, State: Initialized, Role: FOLLOWER
I20260620 11:33:47.769955 11039 tablet_bootstrap.cc:654] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:47.769959 11038 consensus_queue.cc:260] T af48781f875d4f1898eccdfb757ecddb P 64dc8cdf08c5445d9ca51a87379c7df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } }
I20260620 11:33:47.770059 11038 ts_tablet_manager.cc:1428] T af48781f875d4f1898eccdfb757ecddb P 64dc8cdf08c5445d9ca51a87379c7df1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:47.770185 11044 tablet_replica.cc:439] T af48781f875d4f1898eccdfb757ecddb P 64dc8cdf08c5445d9ca51a87379c7df1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } }
I20260620 11:33:47.770285 11044 tablet_replica.cc:442] T af48781f875d4f1898eccdfb757ecddb P 64dc8cdf08c5445d9ca51a87379c7df1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:33:47.770555 11042 tablet_bootstrap.cc:492] T af48781f875d4f1898eccdfb757ecddb P ae7b39323eec4a86bc8f8f27cb547a1d: No bootstrap required, opened a new log
I20260620 11:33:47.770604 11042 ts_tablet_manager.cc:1397] T af48781f875d4f1898eccdfb757ecddb P ae7b39323eec4a86bc8f8f27cb547a1d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:33:47.770728 11039 tablet_bootstrap.cc:492] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5: No bootstrap required, opened a new log
I20260620 11:33:47.770733 11042 raft_consensus.cc:348] T af48781f875d4f1898eccdfb757ecddb P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } }
I20260620 11:33:47.770787 11039 ts_tablet_manager.cc:1397] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:47.770810 11042 raft_consensus.cc:374] T af48781f875d4f1898eccdfb757ecddb P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:47.770834 11042 raft_consensus.cc:729] T af48781f875d4f1898eccdfb757ecddb P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae7b39323eec4a86bc8f8f27cb547a1d, State: Initialized, Role: FOLLOWER
I20260620 11:33:47.770874 11042 consensus_queue.cc:260] T af48781f875d4f1898eccdfb757ecddb P ae7b39323eec4a86bc8f8f27cb547a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } }
I20260620 11:33:47.770992 11042 ts_tablet_manager.cc:1428] T af48781f875d4f1898eccdfb757ecddb P ae7b39323eec4a86bc8f8f27cb547a1d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 11:33:47.770987 11039 raft_consensus.cc:348] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } }
I20260620 11:33:47.771064 11039 raft_consensus.cc:374] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:47.771101 11039 raft_consensus.cc:729] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1ca528ad364d9983417d8b1e442ae5, State: Initialized, Role: FOLLOWER
I20260620 11:33:47.771118 11046 tablet_replica.cc:439] T af48781f875d4f1898eccdfb757ecddb P ae7b39323eec4a86bc8f8f27cb547a1d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } }
I20260620 11:33:47.771163 11039 consensus_queue.cc:260] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } }
I20260620 11:33:47.771189 11046 tablet_replica.cc:442] T af48781f875d4f1898eccdfb757ecddb P ae7b39323eec4a86bc8f8f27cb547a1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:33:47.771274 11039 ts_tablet_manager.cc:1428] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:47.771322 11043 tablet_replica.cc:439] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } }
I20260620 11:33:47.771407 11043 tablet_replica.cc:442] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:33:48.010881 11043 raft_consensus.cc:482] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:33:48.010991 11043 raft_consensus.cc:504] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } }
I20260620 11:33:48.011379 11043 leader_election.cc:288] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64dc8cdf08c5445d9ca51a87379c7df1 (127.2.243.129:36721), ae7b39323eec4a86bc8f8f27cb547a1d (127.2.243.131:43581)
I20260620 11:33:48.014333 11079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:48.011567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:36721 (local address 127.2.243.130:52631)
0620 11:33:48.011707 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:48.011720 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:48.011803 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:48.011958 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:48.011959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:48.011961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:48.012062 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:48.012067 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.012693 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.012694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:48.013504 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:48.013506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.014071 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.014072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:48.014147 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:48.014152 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:33:48.014207 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 11:33:48.014242 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":30,"thread_start_us":60,"threads_started":1}
I20260620 11:33:48.014354 11034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:48.011698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:52631 (local address 127.2.243.129:36721)
0620 11:33:48.011743 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:48.011745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:48.011803 (+ 58us) server_negotiation.cc:407] Connection header received
0620 11:33:48.011890 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:48.011891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:48.011898 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:48.011940 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:48.012135 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.012638 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.013610 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.013992 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.014046 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:48.014053 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:48.014222 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 11:33:48.014263 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:33:48.014448 11080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:48.011552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:43581 (local address 127.2.243.130:42629)
0620 11:33:48.011671 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:48.011683 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:48.011778 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:48.011991 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:48.011992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:48.011994 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:33:48.012062 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:48.012063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.012814 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.012814 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:48.013605 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:48.013607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.014171 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.014171 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:48.014258 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:48.014262 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:33:48.014328 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 11:33:48.014363 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260620 11:33:48.014492 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:48.011688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:42629 (local address 127.2.243.131:43581)
0620 11:33:48.011843 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:48.011845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:48.011854 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:33:48.011884 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:48.011885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:48.011895 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:48.011973 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:33:48.012166 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.012733 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.013691 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.014088 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.014160 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:48.014169 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:48.014353 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 11:33:48.014403 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260620 11:33:48.014665 10963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af48781f875d4f1898eccdfb757ecddb" candidate_uuid: "8f1ca528ad364d9983417d8b1e442ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" is_pre_election: true
I20260620 11:33:48.014763 10963 raft_consensus.cc:2454] T af48781f875d4f1898eccdfb757ecddb P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f1ca528ad364d9983417d8b1e442ae5 in term 0.
I20260620 11:33:48.014758 10682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af48781f875d4f1898eccdfb757ecddb" candidate_uuid: "8f1ca528ad364d9983417d8b1e442ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" is_pre_election: true
I20260620 11:33:48.014835 10682 raft_consensus.cc:2454] T af48781f875d4f1898eccdfb757ecddb P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f1ca528ad364d9983417d8b1e442ae5 in term 0.
I20260620 11:33:48.014937 10758 leader_election.cc:302] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1ca528ad364d9983417d8b1e442ae5, ae7b39323eec4a86bc8f8f27cb547a1d; no voters:
I20260620 11:33:48.015057 11043 raft_consensus.cc:2781] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:33:48.015097 11043 raft_consensus.cc:482] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:33:48.015120 11043 raft_consensus.cc:3037] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:48.015772 11043 raft_consensus.cc:504] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } }
I20260620 11:33:48.015911 11043 leader_election.cc:288] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5 [CANDIDATE]: Term 1 election: Requested vote from peers 64dc8cdf08c5445d9ca51a87379c7df1 (127.2.243.129:36721), ae7b39323eec4a86bc8f8f27cb547a1d (127.2.243.131:43581)
I20260620 11:33:48.016095 10682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af48781f875d4f1898eccdfb757ecddb" candidate_uuid: "8f1ca528ad364d9983417d8b1e442ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64dc8cdf08c5445d9ca51a87379c7df1"
I20260620 11:33:48.016167 10682 raft_consensus.cc:3037] T af48781f875d4f1898eccdfb757ecddb P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:48.016158 10963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af48781f875d4f1898eccdfb757ecddb" candidate_uuid: "8f1ca528ad364d9983417d8b1e442ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d"
I20260620 11:33:48.016258 10963 raft_consensus.cc:3037] T af48781f875d4f1898eccdfb757ecddb P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:48.016755 10682 raft_consensus.cc:2454] T af48781f875d4f1898eccdfb757ecddb P 64dc8cdf08c5445d9ca51a87379c7df1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f1ca528ad364d9983417d8b1e442ae5 in term 1.
I20260620 11:33:48.016863 10963 raft_consensus.cc:2454] T af48781f875d4f1898eccdfb757ecddb P ae7b39323eec4a86bc8f8f27cb547a1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f1ca528ad364d9983417d8b1e442ae5 in term 1.
I20260620 11:33:48.016927 10756 leader_election.cc:302] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64dc8cdf08c5445d9ca51a87379c7df1, 8f1ca528ad364d9983417d8b1e442ae5; no voters:
I20260620 11:33:48.017061 11043 raft_consensus.cc:2781] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:48.017179 11043 raft_consensus.cc:686] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5 [term 1 LEADER]: Becoming Leader. State: Replica: 8f1ca528ad364d9983417d8b1e442ae5, State: Running, Role: LEADER
I20260620 11:33:48.017253 11043 consensus_queue.cc:237] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } }
I20260620 11:33:48.017565 11082 tablet_replica.cc:439] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5: TxnStatusTablet state changed. Reason: New leader 8f1ca528ad364d9983417d8b1e442ae5. Latest consensus state: current_term: 1 leader_uuid: "8f1ca528ad364d9983417d8b1e442ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } }
I20260620 11:33:48.017648 11082 tablet_replica.cc:442] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5: This TxnStatusTablet replica's current role is: LEADER
I20260620 11:33:48.017856 11084 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 11:33:48.017976 11084 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 11:33:48.017926 10526 catalog_manager.cc:5495] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f1ca528ad364d9983417d8b1e442ae5 (127.2.243.130). New cstate: current_term: 1 leader_uuid: "8f1ca528ad364d9983417d8b1e442ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } health_report { overall_health: HEALTHY } } }
I20260620 11:33:48.056239 11037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:48.052974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35690 (local address 127.2.243.130:33403)
0620 11:33:48.053058 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:48.053060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:48.053071 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:33:48.053107 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:48.053109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:48.053128 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:48.053204 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:48.053416 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.054141 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.055288 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.055580 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.055666 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:48.055868 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:48.055966 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:48.056031 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:48.056109 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 11:33:48.056126 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 11:33:48.056260 11074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:48.052876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:33403 (local address 127.0.0.1:35690)
0620 11:33:48.052970 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:48.052984 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:48.053036 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:48.053279 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:48.053280 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:48.053281 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:33:48.053334 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:48.053336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.054244 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.054245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:48.055200 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:48.055203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.055678 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.055679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:48.055749 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:48.055766 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:33:48.056040 (+ 274us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:33:48.056047 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 11:33:48.056105 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 11:33:48.056176 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 11:33:48.057462 10963 raft_consensus.cc:1261] T af48781f875d4f1898eccdfb757ecddb P ae7b39323eec4a86bc8f8f27cb547a1d [term 1 FOLLOWER]: Refusing update from remote peer 8f1ca528ad364d9983417d8b1e442ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:48.057516 10682 raft_consensus.cc:1261] T af48781f875d4f1898eccdfb757ecddb P 64dc8cdf08c5445d9ca51a87379c7df1 [term 1 FOLLOWER]: Refusing update from remote peer 8f1ca528ad364d9983417d8b1e442ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:48.057796 11043 consensus_queue.cc:1035] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:48.057933 11082 consensus_queue.cc:1035] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:48.058508 11046 tablet_replica.cc:439] T af48781f875d4f1898eccdfb757ecddb P ae7b39323eec4a86bc8f8f27cb547a1d: TxnStatusTablet state changed. Reason: New leader 8f1ca528ad364d9983417d8b1e442ae5. Latest consensus state: current_term: 1 leader_uuid: "8f1ca528ad364d9983417d8b1e442ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } }
I20260620 11:33:48.058598 11046 tablet_replica.cc:442] T af48781f875d4f1898eccdfb757ecddb P ae7b39323eec4a86bc8f8f27cb547a1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:33:48.058548 11044 tablet_replica.cc:439] T af48781f875d4f1898eccdfb757ecddb P 64dc8cdf08c5445d9ca51a87379c7df1: TxnStatusTablet state changed. Reason: New leader 8f1ca528ad364d9983417d8b1e442ae5. Latest consensus state: current_term: 1 leader_uuid: "8f1ca528ad364d9983417d8b1e442ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } }
I20260620 11:33:48.058646 11044 tablet_replica.cc:442] T af48781f875d4f1898eccdfb757ecddb P 64dc8cdf08c5445d9ca51a87379c7df1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:33:48.059616 11043 tablet_replica.cc:439] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f1ca528ad364d9983417d8b1e442ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } }
I20260620 11:33:48.059650 11093 tablet_replica.cc:439] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8f1ca528ad364d9983417d8b1e442ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } }
I20260620 11:33:48.059710 11043 tablet_replica.cc:442] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5: This TxnStatusTablet replica's current role is: LEADER
I20260620 11:33:48.059717 11093 tablet_replica.cc:442] T af48781f875d4f1898eccdfb757ecddb P 8f1ca528ad364d9983417d8b1e442ae5: This TxnStatusTablet replica's current role is: LEADER
I20260620 11:33:48.060096 11044 tablet_replica.cc:439] T af48781f875d4f1898eccdfb757ecddb P 64dc8cdf08c5445d9ca51a87379c7df1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f1ca528ad364d9983417d8b1e442ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } }
I20260620 11:33:48.060186 11044 tablet_replica.cc:442] T af48781f875d4f1898eccdfb757ecddb P 64dc8cdf08c5445d9ca51a87379c7df1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:33:48.060382 11046 tablet_replica.cc:439] T af48781f875d4f1898eccdfb757ecddb P ae7b39323eec4a86bc8f8f27cb547a1d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8f1ca528ad364d9983417d8b1e442ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } }
I20260620 11:33:48.060453 11046 tablet_replica.cc:442] T af48781f875d4f1898eccdfb757ecddb P ae7b39323eec4a86bc8f8f27cb547a1d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:33:48.064205 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:48.061585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60606 (local address 127.2.243.131:43581)
0620 11:33:48.061627 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:48.061629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:48.061670 (+ 41us) server_negotiation.cc:407] Connection header received
0620 11:33:48.061741 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:48.061742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:48.061760 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:48.061841 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:48.061918 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.062587 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.063368 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.063667 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.063734 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:48.063866 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:48.063982 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:48.064057 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:48.064119 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 11:33:48.064139 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:33:48.069157 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:48.066418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:33403 (local address 127.2.243.131:34519)
0620 11:33:48.066552 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:48.066562 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:48.066624 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:48.066745 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:48.066746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:48.066747 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:33:48.066792 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:48.066792 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.067676 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.067677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:48.068300 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:48.068302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.068791 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.068792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:48.068840 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:48.068853 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:33:48.069023 (+ 170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:33:48.069025 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:33:48.069059 (+ 34us) client_negotiation.cc:240] Negotiation successful
0620 11:33:48.069095 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":66,"threads_started":1}
I20260620 11:33:48.069156 11037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:48.066512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:34519 (local address 127.2.243.130:33403)
0620 11:33:48.066558 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:48.066560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:48.066626 (+ 66us) server_negotiation.cc:407] Connection header received
0620 11:33:48.066665 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:48.066666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:48.066677 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:48.066734 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:48.066849 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.067575 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.068383 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.068681 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.068752 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:48.068930 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:48.068959 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:48.069013 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:48.069063 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 11:33:48.069081 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:33:48.071564 10824 raft_consensus.cc:1261] T 0268244f10d842e48e2d8dda45b28b66 P 8f1ca528ad364d9983417d8b1e442ae5 [term 1 FOLLOWER]: Refusing update from remote peer ae7b39323eec4a86bc8f8f27cb547a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:48.071576 10682 raft_consensus.cc:1261] T 0268244f10d842e48e2d8dda45b28b66 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 1 FOLLOWER]: Refusing update from remote peer ae7b39323eec4a86bc8f8f27cb547a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:48.071800 11046 consensus_queue.cc:1035] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:48.071916 11067 consensus_queue.cc:1035] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:48.080703 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:48.077754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:59757 (local address 127.2.243.131:43581)
0620 11:33:48.077794 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:48.077796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:48.077933 (+ 137us) server_negotiation.cc:407] Connection header received
0620 11:33:48.077975 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:48.077976 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:48.077984 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:48.078042 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:48.078197 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.078635 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.079962 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.080228 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.080279 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:48.080451 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:48.080514 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:48.080561 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:48.080635 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 11:33:48.080653 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:33:48.080965 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:48.077667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:43581 (local address 127.2.243.130:59757)
0620 11:33:48.077860 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:48.077870 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:48.077930 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:48.078047 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:48.078047 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:48.078048 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:33:48.078127 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:48.078129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.078707 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.078708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:48.079874 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:48.079876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.080293 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.080294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:48.080359 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:48.080372 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:33:48.080579 (+ 207us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:33:48.080581 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:33:48.080632 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 11:33:48.080674 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":94,"threads_started":1}
I20260620 11:33:48.089144 10526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:48.089259 10526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:48.089854 10526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526: Sending DeleteTablet for 3 replicas of tablet 0268244f10d842e48e2d8dda45b28b66
I20260620 11:33:48.090108 10664 tablet_service.cc:1514] Processing DeleteTablet for tablet 0268244f10d842e48e2d8dda45b28b66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:48 UTC) from {username='oryx', principal='oryx/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:51256
I20260620 11:33:48.090158 10944 tablet_service.cc:1514] Processing DeleteTablet for tablet 0268244f10d842e48e2d8dda45b28b66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:48 UTC) from {username='oryx', principal='oryx/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:60602
I20260620 11:33:48.090432 10804 tablet_service.cc:1514] Processing DeleteTablet for tablet 0268244f10d842e48e2d8dda45b28b66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:48 UTC) from {username='oryx', principal='oryx/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:35682
I20260620 11:33:48.090708 11108 tablet_replica.cc:330] T 0268244f10d842e48e2d8dda45b28b66 P 64dc8cdf08c5445d9ca51a87379c7df1: stopping tablet replica
I20260620 11:33:48.090832 11110 tablet_replica.cc:330] T 0268244f10d842e48e2d8dda45b28b66 P 8f1ca528ad364d9983417d8b1e442ae5: stopping tablet replica
I20260620 11:33:48.090942 11108 raft_consensus.cc:2229] T 0268244f10d842e48e2d8dda45b28b66 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:48.091041 11110 raft_consensus.cc:2229] T 0268244f10d842e48e2d8dda45b28b66 P 8f1ca528ad364d9983417d8b1e442ae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:48.091135 11108 raft_consensus.cc:2258] T 0268244f10d842e48e2d8dda45b28b66 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:48.091215 11110 raft_consensus.cc:2258] T 0268244f10d842e48e2d8dda45b28b66 P 8f1ca528ad364d9983417d8b1e442ae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:48.091625 11108 ts_tablet_manager.cc:1905] T 0268244f10d842e48e2d8dda45b28b66 P 64dc8cdf08c5445d9ca51a87379c7df1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:48.091877 11110 ts_tablet_manager.cc:1905] T 0268244f10d842e48e2d8dda45b28b66 P 8f1ca528ad364d9983417d8b1e442ae5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:48.093555 11109 tablet_replica.cc:330] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d: stopping tablet replica
I20260620 11:33:48.093621 11109 raft_consensus.cc:2229] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:33:48.093704 11109 raft_consensus.cc:2258] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:48.093941 11109 ts_tablet_manager.cc:1905] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:48.095389 11108 ts_tablet_manager.cc:1918] T 0268244f10d842e48e2d8dda45b28b66 P 64dc8cdf08c5445d9ca51a87379c7df1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:33:48.095511 11109 ts_tablet_manager.cc:1918] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:33:48.095570 11109 log.cc:1196] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/0268244f10d842e48e2d8dda45b28b66
I20260620 11:33:48.095820 11109 ts_tablet_manager.cc:1939] T 0268244f10d842e48e2d8dda45b28b66 P ae7b39323eec4a86bc8f8f27cb547a1d: Deleting consensus metadata
I20260620 11:33:48.095927 11108 log.cc:1196] T 0268244f10d842e48e2d8dda45b28b66 P 64dc8cdf08c5445d9ca51a87379c7df1: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-0/wal/wals/0268244f10d842e48e2d8dda45b28b66
I20260620 11:33:48.096148 11108 ts_tablet_manager.cc:1939] T 0268244f10d842e48e2d8dda45b28b66 P 64dc8cdf08c5445d9ca51a87379c7df1: Deleting consensus metadata
I20260620 11:33:48.096503 10512 catalog_manager.cc:4841] TS 64dc8cdf08c5445d9ca51a87379c7df1 (127.2.243.129:36721): tablet 0268244f10d842e48e2d8dda45b28b66 (table test-table [id=68cd03faa28f47d4af27175a18987b41]) successfully deleted
I20260620 11:33:48.096554 11110 ts_tablet_manager.cc:1918] T 0268244f10d842e48e2d8dda45b28b66 P 8f1ca528ad364d9983417d8b1e442ae5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:33:48.096604 11110 log.cc:1196] T 0268244f10d842e48e2d8dda45b28b66 P 8f1ca528ad364d9983417d8b1e442ae5: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-1/wal/wals/0268244f10d842e48e2d8dda45b28b66
I20260620 11:33:48.096807 11110 ts_tablet_manager.cc:1939] T 0268244f10d842e48e2d8dda45b28b66 P 8f1ca528ad364d9983417d8b1e442ae5: Deleting consensus metadata
I20260620 11:33:48.097147 10514 catalog_manager.cc:4841] TS 8f1ca528ad364d9983417d8b1e442ae5 (127.2.243.130:33403): tablet 0268244f10d842e48e2d8dda45b28b66 (table test-table [id=68cd03faa28f47d4af27175a18987b41]) successfully deleted
I20260620 11:33:48.100396 10514 catalog_manager.cc:4841] TS ae7b39323eec4a86bc8f8f27cb547a1d (127.2.243.131:43581): tablet 0268244f10d842e48e2d8dda45b28b66 (table test-table [id=68cd03faa28f47d4af27175a18987b41]) successfully deleted
Jun 20 11:33:48 dist-test-slave-ctbw krb5kdc[10482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 11:33:48 dist-test-slave-ctbw krb5kdc[10482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:48 dist-test-slave-ctbw krb5kdc[10482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 11:33:48.120340 11075 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:48.114928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50094 (local address 127.2.243.190:33311)
0620 11:33:48.114979 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:48.114982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:48.115103 (+ 121us) server_negotiation.cc:407] Connection header received
0620 11:33:48.115193 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:48.115194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:48.115204 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:48.115280 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:48.115890 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.116387 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.117131 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.117332 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.117417 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:48.120289 (+ 2872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50094: BlockingRecv error: recv got EOF from 127.0.0.1:50094 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 11:33:48 dist-test-slave-ctbw krb5kdc[10482](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM
I20260620 11:33:48.128453 11075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:48.121308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50098 (local address 127.2.243.190:33311)
0620 11:33:48.121347 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:48.121349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:48.121555 (+ 206us) server_negotiation.cc:407] Connection header received
0620 11:33:48.121661 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:48.121662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:48.121673 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:48.121718 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:48.122151 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.122615 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.123351 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.123553 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.123634 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:48.125785 (+ 2151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:48.125786 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:48.125787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:48.125795 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:48.127322 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:48.127670 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:48.127671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:48.127671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:48.127729 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:48.127930 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:48.127931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:48.127932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:48.128039 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:48.128147 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:48.128241 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 11:33:48.128324 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:33:48.130098 10526 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:33:48.130326 10526 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:33:48.131678 10944 tablet_service.cc:1467] Processing CreateTablet for tablet 6b98b3452c3146f5b826cdf66eaabe49 (DEFAULT_TABLE table=test-table [id=3f4aad686ca348e4adde2a9876ed8b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:48.131678 10664 tablet_service.cc:1467] Processing CreateTablet for tablet 6b98b3452c3146f5b826cdf66eaabe49 (DEFAULT_TABLE table=test-table [id=3f4aad686ca348e4adde2a9876ed8b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:48.131824 10664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b98b3452c3146f5b826cdf66eaabe49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:48.131824 10944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b98b3452c3146f5b826cdf66eaabe49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:48.133165 11042 tablet_bootstrap.cc:492] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d: Bootstrap starting.
I20260620 11:33:48.133162 11038 tablet_bootstrap.cc:492] T 6b98b3452c3146f5b826cdf66eaabe49 P 64dc8cdf08c5445d9ca51a87379c7df1: Bootstrap starting.
I20260620 11:33:48.133559 11038 tablet_bootstrap.cc:654] T 6b98b3452c3146f5b826cdf66eaabe49 P 64dc8cdf08c5445d9ca51a87379c7df1: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:48.133575 11042 tablet_bootstrap.cc:654] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:48.134022 10804 tablet_service.cc:1467] Processing CreateTablet for tablet 6b98b3452c3146f5b826cdf66eaabe49 (DEFAULT_TABLE table=test-table [id=3f4aad686ca348e4adde2a9876ed8b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:48.134150 10804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b98b3452c3146f5b826cdf66eaabe49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:48.134683 11042 tablet_bootstrap.cc:492] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d: No bootstrap required, opened a new log
I20260620 11:33:48.134729 11042 ts_tablet_manager.cc:1397] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 11:33:48.134905 11042 raft_consensus.cc:348] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:48.134981 11042 raft_consensus.cc:374] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:48.135054 11042 raft_consensus.cc:729] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae7b39323eec4a86bc8f8f27cb547a1d, State: Initialized, Role: FOLLOWER
I20260620 11:33:48.135105 11042 consensus_queue.cc:260] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:48.135218 11042 ts_tablet_manager.cc:1428] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:48.135828 11039 tablet_bootstrap.cc:492] T 6b98b3452c3146f5b826cdf66eaabe49 P 8f1ca528ad364d9983417d8b1e442ae5: Bootstrap starting.
I20260620 11:33:48.136269 11039 tablet_bootstrap.cc:654] T 6b98b3452c3146f5b826cdf66eaabe49 P 8f1ca528ad364d9983417d8b1e442ae5: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:48.136859 11039 tablet_bootstrap.cc:492] T 6b98b3452c3146f5b826cdf66eaabe49 P 8f1ca528ad364d9983417d8b1e442ae5: No bootstrap required, opened a new log
I20260620 11:33:48.136924 11039 ts_tablet_manager.cc:1397] T 6b98b3452c3146f5b826cdf66eaabe49 P 8f1ca528ad364d9983417d8b1e442ae5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:48.137302 11039 raft_consensus.cc:348] T 6b98b3452c3146f5b826cdf66eaabe49 P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:48.137379 11039 raft_consensus.cc:374] T 6b98b3452c3146f5b826cdf66eaabe49 P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:48.137408 11039 raft_consensus.cc:729] T 6b98b3452c3146f5b826cdf66eaabe49 P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1ca528ad364d9983417d8b1e442ae5, State: Initialized, Role: FOLLOWER
I20260620 11:33:48.137449 11039 consensus_queue.cc:260] T 6b98b3452c3146f5b826cdf66eaabe49 P 8f1ca528ad364d9983417d8b1e442ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:48.137545 11039 ts_tablet_manager.cc:1428] T 6b98b3452c3146f5b826cdf66eaabe49 P 8f1ca528ad364d9983417d8b1e442ae5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:48.138003 11038 tablet_bootstrap.cc:492] T 6b98b3452c3146f5b826cdf66eaabe49 P 64dc8cdf08c5445d9ca51a87379c7df1: No bootstrap required, opened a new log
I20260620 11:33:48.138060 11038 ts_tablet_manager.cc:1397] T 6b98b3452c3146f5b826cdf66eaabe49 P 64dc8cdf08c5445d9ca51a87379c7df1: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260620 11:33:48.138170 11038 raft_consensus.cc:348] T 6b98b3452c3146f5b826cdf66eaabe49 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:48.138255 11038 raft_consensus.cc:374] T 6b98b3452c3146f5b826cdf66eaabe49 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:48.138283 11038 raft_consensus.cc:729] T 6b98b3452c3146f5b826cdf66eaabe49 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64dc8cdf08c5445d9ca51a87379c7df1, State: Initialized, Role: FOLLOWER
I20260620 11:33:48.138322 11038 consensus_queue.cc:260] T 6b98b3452c3146f5b826cdf66eaabe49 P 64dc8cdf08c5445d9ca51a87379c7df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:48.138419 11038 ts_tablet_manager.cc:1428] T 6b98b3452c3146f5b826cdf66eaabe49 P 64dc8cdf08c5445d9ca51a87379c7df1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:48.340656 11046 raft_consensus.cc:482] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:33:48.340757 11046 raft_consensus.cc:504] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:48.340963 11046 leader_election.cc:288] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 64dc8cdf08c5445d9ca51a87379c7df1 (127.2.243.129:36721), 8f1ca528ad364d9983417d8b1e442ae5 (127.2.243.130:33403)
I20260620 11:33:48.341251 10824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b98b3452c3146f5b826cdf66eaabe49" candidate_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1ca528ad364d9983417d8b1e442ae5" is_pre_election: true
I20260620 11:33:48.341251 10682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b98b3452c3146f5b826cdf66eaabe49" candidate_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" is_pre_election: true
I20260620 11:33:48.341365 10682 raft_consensus.cc:2454] T 6b98b3452c3146f5b826cdf66eaabe49 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae7b39323eec4a86bc8f8f27cb547a1d in term 0.
I20260620 11:33:48.341365 10824 raft_consensus.cc:2454] T 6b98b3452c3146f5b826cdf66eaabe49 P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae7b39323eec4a86bc8f8f27cb547a1d in term 0.
I20260620 11:33:48.341534 10898 leader_election.cc:302] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1ca528ad364d9983417d8b1e442ae5, ae7b39323eec4a86bc8f8f27cb547a1d; no voters:
I20260620 11:33:48.341643 11046 raft_consensus.cc:2781] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:33:48.341687 11046 raft_consensus.cc:482] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:33:48.341708 11046 raft_consensus.cc:3037] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:48.342402 11046 raft_consensus.cc:504] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:48.342549 11046 leader_election.cc:288] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d [CANDIDATE]: Term 1 election: Requested vote from peers 64dc8cdf08c5445d9ca51a87379c7df1 (127.2.243.129:36721), 8f1ca528ad364d9983417d8b1e442ae5 (127.2.243.130:33403)
I20260620 11:33:48.342715 10682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b98b3452c3146f5b826cdf66eaabe49" candidate_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64dc8cdf08c5445d9ca51a87379c7df1"
I20260620 11:33:48.342711 10824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b98b3452c3146f5b826cdf66eaabe49" candidate_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1ca528ad364d9983417d8b1e442ae5"
I20260620 11:33:48.342785 10682 raft_consensus.cc:3037] T 6b98b3452c3146f5b826cdf66eaabe49 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:48.342787 10824 raft_consensus.cc:3037] T 6b98b3452c3146f5b826cdf66eaabe49 P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:48.343542 10824 raft_consensus.cc:2454] T 6b98b3452c3146f5b826cdf66eaabe49 P 8f1ca528ad364d9983417d8b1e442ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae7b39323eec4a86bc8f8f27cb547a1d in term 1.
I20260620 11:33:48.343557 10682 raft_consensus.cc:2454] T 6b98b3452c3146f5b826cdf66eaabe49 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae7b39323eec4a86bc8f8f27cb547a1d in term 1.
I20260620 11:33:48.343722 10898 leader_election.cc:302] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f1ca528ad364d9983417d8b1e442ae5, ae7b39323eec4a86bc8f8f27cb547a1d; no voters:
I20260620 11:33:48.343840 11046 raft_consensus.cc:2781] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:48.343897 11046 raft_consensus.cc:686] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d [term 1 LEADER]: Becoming Leader. State: Replica: ae7b39323eec4a86bc8f8f27cb547a1d, State: Running, Role: LEADER
I20260620 11:33:48.343950 11046 consensus_queue.cc:237] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:48.344573 10526 catalog_manager.cc:5495] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d reported cstate change: term changed from 0 to 1, leader changed from <none> to ae7b39323eec4a86bc8f8f27cb547a1d (127.2.243.131). New cstate: current_term: 1 leader_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } health_report { overall_health: HEALTHY } } }
I20260620 11:33:48.362303 11081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:48.360052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60614 (local address 127.2.243.131:43581)
0620 11:33:48.360115 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:48.360117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:48.360125 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:33:48.360147 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:48.360148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:48.360158 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:48.360201 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:48.360348 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.360805 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.361597 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.361808 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.361884 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:48.362049 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:48.362083 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:48.362135 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:48.362201 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 11:33:48.362220 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 11:33:48.363104 10824 raft_consensus.cc:1261] T 6b98b3452c3146f5b826cdf66eaabe49 P 8f1ca528ad364d9983417d8b1e442ae5 [term 1 FOLLOWER]: Refusing update from remote peer ae7b39323eec4a86bc8f8f27cb547a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:48.363126 10682 raft_consensus.cc:1261] T 6b98b3452c3146f5b826cdf66eaabe49 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 1 FOLLOWER]: Refusing update from remote peer ae7b39323eec4a86bc8f8f27cb547a1d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:48.363350 11067 consensus_queue.cc:1035] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:48.363477 11046 consensus_queue.cc:1035] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:48.366230 10527 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:48.366313 10527 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:48.366956 10527 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526: Sending DeleteTablet for 3 replicas of tablet 6b98b3452c3146f5b826cdf66eaabe49
I20260620 11:33:48.367211 10664 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b98b3452c3146f5b826cdf66eaabe49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:48 UTC) from {username='oryx', principal='oryx/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:51256
I20260620 11:33:48.367213 10944 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b98b3452c3146f5b826cdf66eaabe49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:48 UTC) from {username='oryx', principal='oryx/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:60602
I20260620 11:33:48.367259 10804 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b98b3452c3146f5b826cdf66eaabe49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:48 UTC) from {username='oryx', principal='oryx/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:35682
I20260620 11:33:48.367295 11108 tablet_replica.cc:330] T 6b98b3452c3146f5b826cdf66eaabe49 P 64dc8cdf08c5445d9ca51a87379c7df1: stopping tablet replica
I20260620 11:33:48.367295 11109 tablet_replica.cc:330] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d: stopping tablet replica
I20260620 11:33:48.367342 11108 raft_consensus.cc:2229] T 6b98b3452c3146f5b826cdf66eaabe49 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:48.367339 11109 raft_consensus.cc:2229] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:33:48.367352 11110 tablet_replica.cc:330] T 6b98b3452c3146f5b826cdf66eaabe49 P 8f1ca528ad364d9983417d8b1e442ae5: stopping tablet replica
I20260620 11:33:48.367378 11108 raft_consensus.cc:2258] T 6b98b3452c3146f5b826cdf66eaabe49 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:48.367393 11109 raft_consensus.cc:2258] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:48.367401 11110 raft_consensus.cc:2229] T 6b98b3452c3146f5b826cdf66eaabe49 P 8f1ca528ad364d9983417d8b1e442ae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:48.367456 11110 raft_consensus.cc:2258] T 6b98b3452c3146f5b826cdf66eaabe49 P 8f1ca528ad364d9983417d8b1e442ae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:48.367691 11109 ts_tablet_manager.cc:1905] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:48.367697 11108 ts_tablet_manager.cc:1905] T 6b98b3452c3146f5b826cdf66eaabe49 P 64dc8cdf08c5445d9ca51a87379c7df1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:48.367655 10527 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50098:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:33:48.367784 10527 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:33:48.368186 11110 ts_tablet_manager.cc:1905] T 6b98b3452c3146f5b826cdf66eaabe49 P 8f1ca528ad364d9983417d8b1e442ae5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:48.368852 10664 tablet_service.cc:1467] Processing CreateTablet for tablet 6ad5449b675741bc97f221c0b0a42bc1 (DEFAULT_TABLE table=test-table [id=0cbafd08ec0f4d869e70749170474da2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:48.368947 10664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ad5449b675741bc97f221c0b0a42bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:48.369342 11108 ts_tablet_manager.cc:1918] T 6b98b3452c3146f5b826cdf66eaabe49 P 64dc8cdf08c5445d9ca51a87379c7df1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:33:48.369354 11109 ts_tablet_manager.cc:1918] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:33:48.369397 11108 log.cc:1196] T 6b98b3452c3146f5b826cdf66eaabe49 P 64dc8cdf08c5445d9ca51a87379c7df1: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-0/wal/wals/6b98b3452c3146f5b826cdf66eaabe49
I20260620 11:33:48.369402 11109 log.cc:1196] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/6b98b3452c3146f5b826cdf66eaabe49
I20260620 11:33:48.369634 11109 ts_tablet_manager.cc:1939] T 6b98b3452c3146f5b826cdf66eaabe49 P ae7b39323eec4a86bc8f8f27cb547a1d: Deleting consensus metadata
I20260620 11:33:48.369685 11108 ts_tablet_manager.cc:1939] T 6b98b3452c3146f5b826cdf66eaabe49 P 64dc8cdf08c5445d9ca51a87379c7df1: Deleting consensus metadata
I20260620 11:33:48.369869 11110 ts_tablet_manager.cc:1918] T 6b98b3452c3146f5b826cdf66eaabe49 P 8f1ca528ad364d9983417d8b1e442ae5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:33:48.369920 11110 log.cc:1196] T 6b98b3452c3146f5b826cdf66eaabe49 P 8f1ca528ad364d9983417d8b1e442ae5: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-1/wal/wals/6b98b3452c3146f5b826cdf66eaabe49
I20260620 11:33:48.370168 11110 ts_tablet_manager.cc:1939] T 6b98b3452c3146f5b826cdf66eaabe49 P 8f1ca528ad364d9983417d8b1e442ae5: Deleting consensus metadata
I20260620 11:33:48.370327 11038 tablet_bootstrap.cc:492] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1: Bootstrap starting.
I20260620 11:33:48.370393 10512 catalog_manager.cc:4841] TS 64dc8cdf08c5445d9ca51a87379c7df1 (127.2.243.129:36721): tablet 6b98b3452c3146f5b826cdf66eaabe49 (table test-table [id=3f4aad686ca348e4adde2a9876ed8b79]) successfully deleted
I20260620 11:33:48.370518 10514 catalog_manager.cc:4841] TS 8f1ca528ad364d9983417d8b1e442ae5 (127.2.243.130:33403): tablet 6b98b3452c3146f5b826cdf66eaabe49 (table test-table [id=3f4aad686ca348e4adde2a9876ed8b79]) successfully deleted
I20260620 11:33:48.370575 10514 catalog_manager.cc:4841] TS ae7b39323eec4a86bc8f8f27cb547a1d (127.2.243.131:43581): tablet 6b98b3452c3146f5b826cdf66eaabe49 (table test-table [id=3f4aad686ca348e4adde2a9876ed8b79]) successfully deleted
I20260620 11:33:48.370596 10944 tablet_service.cc:1467] Processing CreateTablet for tablet 6ad5449b675741bc97f221c0b0a42bc1 (DEFAULT_TABLE table=test-table [id=0cbafd08ec0f4d869e70749170474da2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:48.370673 11038 tablet_bootstrap.cc:654] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:48.370689 10944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ad5449b675741bc97f221c0b0a42bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:48.370720 10804 tablet_service.cc:1467] Processing CreateTablet for tablet 6ad5449b675741bc97f221c0b0a42bc1 (DEFAULT_TABLE table=test-table [id=0cbafd08ec0f4d869e70749170474da2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:33:48.370826 10804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ad5449b675741bc97f221c0b0a42bc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:48.371784 11042 tablet_bootstrap.cc:492] T 6ad5449b675741bc97f221c0b0a42bc1 P ae7b39323eec4a86bc8f8f27cb547a1d: Bootstrap starting.
I20260620 11:33:48.371956 11039 tablet_bootstrap.cc:492] T 6ad5449b675741bc97f221c0b0a42bc1 P 8f1ca528ad364d9983417d8b1e442ae5: Bootstrap starting.
I20260620 11:33:48.372066 11042 tablet_bootstrap.cc:654] T 6ad5449b675741bc97f221c0b0a42bc1 P ae7b39323eec4a86bc8f8f27cb547a1d: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:48.372303 11039 tablet_bootstrap.cc:654] T 6ad5449b675741bc97f221c0b0a42bc1 P 8f1ca528ad364d9983417d8b1e442ae5: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:48.372310 11038 tablet_bootstrap.cc:492] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1: No bootstrap required, opened a new log
I20260620 11:33:48.372350 11038 ts_tablet_manager.cc:1397] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:33:48.372435 11038 raft_consensus.cc:348] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:48.372476 11038 raft_consensus.cc:374] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:48.372511 11038 raft_consensus.cc:729] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64dc8cdf08c5445d9ca51a87379c7df1, State: Initialized, Role: FOLLOWER
I20260620 11:33:48.372556 11038 consensus_queue.cc:260] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:48.372659 11042 tablet_bootstrap.cc:492] T 6ad5449b675741bc97f221c0b0a42bc1 P ae7b39323eec4a86bc8f8f27cb547a1d: No bootstrap required, opened a new log
I20260620 11:33:48.372666 11038 ts_tablet_manager.cc:1428] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:48.372704 11042 ts_tablet_manager.cc:1397] T 6ad5449b675741bc97f221c0b0a42bc1 P ae7b39323eec4a86bc8f8f27cb547a1d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 11:33:48.372820 11042 raft_consensus.cc:348] T 6ad5449b675741bc97f221c0b0a42bc1 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:48.372922 11042 raft_consensus.cc:374] T 6ad5449b675741bc97f221c0b0a42bc1 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:48.372956 11042 raft_consensus.cc:729] T 6ad5449b675741bc97f221c0b0a42bc1 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae7b39323eec4a86bc8f8f27cb547a1d, State: Initialized, Role: FOLLOWER
I20260620 11:33:48.372991 11042 consensus_queue.cc:260] T 6ad5449b675741bc97f221c0b0a42bc1 P ae7b39323eec4a86bc8f8f27cb547a1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:48.373085 11042 ts_tablet_manager.cc:1428] T 6ad5449b675741bc97f221c0b0a42bc1 P ae7b39323eec4a86bc8f8f27cb547a1d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:48.373452 11039 tablet_bootstrap.cc:492] T 6ad5449b675741bc97f221c0b0a42bc1 P 8f1ca528ad364d9983417d8b1e442ae5: No bootstrap required, opened a new log
I20260620 11:33:48.373492 11039 ts_tablet_manager.cc:1397] T 6ad5449b675741bc97f221c0b0a42bc1 P 8f1ca528ad364d9983417d8b1e442ae5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260620 11:33:48.373610 11039 raft_consensus.cc:348] T 6ad5449b675741bc97f221c0b0a42bc1 P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:48.373669 11039 raft_consensus.cc:374] T 6ad5449b675741bc97f221c0b0a42bc1 P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:48.373683 11039 raft_consensus.cc:729] T 6ad5449b675741bc97f221c0b0a42bc1 P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f1ca528ad364d9983417d8b1e442ae5, State: Initialized, Role: FOLLOWER
I20260620 11:33:48.373733 11039 consensus_queue.cc:260] T 6ad5449b675741bc97f221c0b0a42bc1 P 8f1ca528ad364d9983417d8b1e442ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:48.373826 11039 ts_tablet_manager.cc:1428] T 6ad5449b675741bc97f221c0b0a42bc1 P 8f1ca528ad364d9983417d8b1e442ae5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:48.428212 11044 raft_consensus.cc:482] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:33:48.428313 11044 raft_consensus.cc:504] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:48.428524 11044 leader_election.cc:288] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f1ca528ad364d9983417d8b1e442ae5 (127.2.243.130:33403), ae7b39323eec4a86bc8f8f27cb547a1d (127.2.243.131:43581)
I20260620 11:33:48.428737 10824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad5449b675741bc97f221c0b0a42bc1" candidate_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1ca528ad364d9983417d8b1e442ae5" is_pre_election: true
I20260620 11:33:48.428792 10963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad5449b675741bc97f221c0b0a42bc1" candidate_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" is_pre_election: true
I20260620 11:33:48.428874 10824 raft_consensus.cc:2454] T 6ad5449b675741bc97f221c0b0a42bc1 P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64dc8cdf08c5445d9ca51a87379c7df1 in term 0.
I20260620 11:33:48.428874 10963 raft_consensus.cc:2454] T 6ad5449b675741bc97f221c0b0a42bc1 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64dc8cdf08c5445d9ca51a87379c7df1 in term 0.
I20260620 11:33:48.429021 10618 leader_election.cc:302] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64dc8cdf08c5445d9ca51a87379c7df1, 8f1ca528ad364d9983417d8b1e442ae5; no voters:
I20260620 11:33:48.429147 11044 raft_consensus.cc:2781] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:33:48.429186 11044 raft_consensus.cc:482] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:33:48.429212 11044 raft_consensus.cc:3037] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:48.430044 11044 raft_consensus.cc:504] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:48.430179 11044 leader_election.cc:288] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1 [CANDIDATE]: Term 1 election: Requested vote from peers 8f1ca528ad364d9983417d8b1e442ae5 (127.2.243.130:33403), ae7b39323eec4a86bc8f8f27cb547a1d (127.2.243.131:43581)
I20260620 11:33:48.430312 10824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad5449b675741bc97f221c0b0a42bc1" candidate_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f1ca528ad364d9983417d8b1e442ae5"
I20260620 11:33:48.430405 10824 raft_consensus.cc:3037] T 6ad5449b675741bc97f221c0b0a42bc1 P 8f1ca528ad364d9983417d8b1e442ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:48.430364 10963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad5449b675741bc97f221c0b0a42bc1" candidate_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d"
I20260620 11:33:48.430491 10963 raft_consensus.cc:3037] T 6ad5449b675741bc97f221c0b0a42bc1 P ae7b39323eec4a86bc8f8f27cb547a1d [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:48.431087 10824 raft_consensus.cc:2454] T 6ad5449b675741bc97f221c0b0a42bc1 P 8f1ca528ad364d9983417d8b1e442ae5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64dc8cdf08c5445d9ca51a87379c7df1 in term 1.
I20260620 11:33:48.431129 10963 raft_consensus.cc:2454] T 6ad5449b675741bc97f221c0b0a42bc1 P ae7b39323eec4a86bc8f8f27cb547a1d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64dc8cdf08c5445d9ca51a87379c7df1 in term 1.
I20260620 11:33:48.431265 10618 leader_election.cc:302] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64dc8cdf08c5445d9ca51a87379c7df1, ae7b39323eec4a86bc8f8f27cb547a1d; no voters:
I20260620 11:33:48.431391 11044 raft_consensus.cc:2781] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:48.431562 11044 raft_consensus.cc:686] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 1 LEADER]: Becoming Leader. State: Replica: 64dc8cdf08c5445d9ca51a87379c7df1, State: Running, Role: LEADER
I20260620 11:33:48.431635 11044 consensus_queue.cc:237] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } }
I20260620 11:33:48.432199 10526 catalog_manager.cc:5495] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64dc8cdf08c5445d9ca51a87379c7df1 (127.2.243.129). New cstate: current_term: 1 leader_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64dc8cdf08c5445d9ca51a87379c7df1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 36721 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 } health_report { overall_health: UNKNOWN } } }
I20260620 11:33:48.450536 11034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:48.448627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51280 (local address 127.2.243.129:36721)
0620 11:33:48.448681 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:48.448683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:48.448691 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:33:48.448712 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:48.448712 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:48.448722 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:48.448779 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:48.448863 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.449321 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.450055 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.450227 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.450282 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:48.450383 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:48.450406 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:48.450437 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:48.450467 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 11:33:48.450481 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 11:33:48.454936 11037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:48.452793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:55405 (local address 127.2.243.130:33403)
0620 11:33:48.452849 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:48.452851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:48.452987 (+ 136us) server_negotiation.cc:407] Connection header received
0620 11:33:48.453007 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:48.453007 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:48.453015 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:48.453061 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:48.453192 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.453621 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.454313 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.454544 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.454599 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:48.454742 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:48.454765 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:48.454802 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:48.454854 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 11:33:48.454882 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 11:33:48.455001 11138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:48.452739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:33403 (local address 127.2.243.129:55405)
0620 11:33:48.452897 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:48.452907 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:48.452955 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:48.453073 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:48.453074 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:48.453075 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:33:48.453123 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:48.453123 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.453675 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.453676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:48.454232 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:48.454234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.454610 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.454611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:48.454661 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:48.454674 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:33:48.454814 (+ 140us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:33:48.454816 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:33:48.454847 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 11:33:48.454941 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":93,"threads_started":1}
I20260620 11:33:48.457033 10963 raft_consensus.cc:1261] T 6ad5449b675741bc97f221c0b0a42bc1 P ae7b39323eec4a86bc8f8f27cb547a1d [term 1 FOLLOWER]: Refusing update from remote peer 64dc8cdf08c5445d9ca51a87379c7df1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:48.457093 10824 raft_consensus.cc:1261] T 6ad5449b675741bc97f221c0b0a42bc1 P 8f1ca528ad364d9983417d8b1e442ae5 [term 1 FOLLOWER]: Refusing update from remote peer 64dc8cdf08c5445d9ca51a87379c7df1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:33:48.457342 11044 consensus_queue.cc:1035] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f1ca528ad364d9983417d8b1e442ae5" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 33403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:48.457456 11134 consensus_queue.cc:1035] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae7b39323eec4a86bc8f8f27cb547a1d" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:33:48.464380 11034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:48.462256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:35575 (local address 127.2.243.129:36721)
0620 11:33:48.462293 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:48.462295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:48.462305 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:33:48.462365 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:48.462366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:48.462376 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:48.462424 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:33:48.462518 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.463011 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.463734 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.463941 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.464017 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:48.464162 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:33:48.464192 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:33:48.464238 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:48.464294 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 11:33:48.464312 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:33:48.464406 11105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:48.462214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:36721 (local address 127.2.243.130:35575)
0620 11:33:48.462268 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:48.462278 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:48.462313 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:48.462425 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:48.462426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:48.462427 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:33:48.462469 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:48.462471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.463095 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.463096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:48.463639 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:48.463641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:48.464014 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:48.464015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:48.464068 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:48.464074 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:33:48.464238 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:33:48.464240 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:33:48.464290 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 11:33:48.464332 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 11:33:48.470224 10526 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:48.470321 10526 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50098:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:33:48.470929 10526 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 676204a6b302407493c222f7219cd526: Sending DeleteTablet for 3 replicas of tablet 6ad5449b675741bc97f221c0b0a42bc1
I20260620 11:33:48.471181 10664 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ad5449b675741bc97f221c0b0a42bc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:48 UTC) from {username='oryx', principal='oryx/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:51256
I20260620 11:33:48.471195 10944 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ad5449b675741bc97f221c0b0a42bc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:48 UTC) from {username='oryx', principal='oryx/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:60602
I20260620 11:33:48.471272 11109 tablet_replica.cc:330] T 6ad5449b675741bc97f221c0b0a42bc1 P ae7b39323eec4a86bc8f8f27cb547a1d: stopping tablet replica
I20260620 11:33:48.471272 11108 tablet_replica.cc:330] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1: stopping tablet replica
I20260620 11:33:48.471318 11108 raft_consensus.cc:2229] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:33:48.471318 11109 raft_consensus.cc:2229] T 6ad5449b675741bc97f221c0b0a42bc1 P ae7b39323eec4a86bc8f8f27cb547a1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:48.471349 11109 raft_consensus.cc:2258] T 6ad5449b675741bc97f221c0b0a42bc1 P ae7b39323eec4a86bc8f8f27cb547a1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:48.471371 11108 raft_consensus.cc:2258] T 6ad5449b675741bc97f221c0b0a42bc1 P 64dc8cdf08c5445d9ca51a87379c7df1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:48.471516 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 10599
I20260620 11:33:48.471554 10804 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ad5449b675741bc97f221c0b0a42bc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:33:48 UTC) from {username='oryx', principal='oryx/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:35682
I20260620 11:33:48.472016 11110 tablet_replica.cc:330] T 6ad5449b675741bc97f221c0b0a42bc1 P 8f1ca528ad364d9983417d8b1e442ae5: stopping tablet replica
I20260620 11:33:48.472074 11110 raft_consensus.cc:2229] T 6ad5449b675741bc97f221c0b0a42bc1 P 8f1ca528ad364d9983417d8b1e442ae5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:33:48.472106 11110 raft_consensus.cc:2258] T 6ad5449b675741bc97f221c0b0a42bc1 P 8f1ca528ad364d9983417d8b1e442ae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:33:48.472437 11110 ts_tablet_manager.cc:1905] T 6ad5449b675741bc97f221c0b0a42bc1 P 8f1ca528ad364d9983417d8b1e442ae5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:48.473685 11110 ts_tablet_manager.cc:1918] T 6ad5449b675741bc97f221c0b0a42bc1 P 8f1ca528ad364d9983417d8b1e442ae5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:33:48.473733 11110 log.cc:1196] T 6ad5449b675741bc97f221c0b0a42bc1 P 8f1ca528ad364d9983417d8b1e442ae5: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-1/wal/wals/6ad5449b675741bc97f221c0b0a42bc1
I20260620 11:33:48.473941 11110 ts_tablet_manager.cc:1939] T 6ad5449b675741bc97f221c0b0a42bc1 P 8f1ca528ad364d9983417d8b1e442ae5: Deleting consensus metadata
I20260620 11:33:48.473964 11109 ts_tablet_manager.cc:1905] T 6ad5449b675741bc97f221c0b0a42bc1 P ae7b39323eec4a86bc8f8f27cb547a1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:33:48.474252 10514 catalog_manager.cc:4841] TS 8f1ca528ad364d9983417d8b1e442ae5 (127.2.243.130:33403): tablet 6ad5449b675741bc97f221c0b0a42bc1 (table test-table [id=0cbafd08ec0f4d869e70749170474da2]) successfully deleted
I20260620 11:33:48.475188 11109 ts_tablet_manager.cc:1918] T 6ad5449b675741bc97f221c0b0a42bc1 P ae7b39323eec4a86bc8f8f27cb547a1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:33:48.475239 11109 log.cc:1196] T 6ad5449b675741bc97f221c0b0a42bc1 P ae7b39323eec4a86bc8f8f27cb547a1d: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/6ad5449b675741bc97f221c0b0a42bc1
I20260620 11:33:48.475437 11109 ts_tablet_manager.cc:1939] T 6ad5449b675741bc97f221c0b0a42bc1 P ae7b39323eec4a86bc8f8f27cb547a1d: Deleting consensus metadata
I20260620 11:33:48.475759 10514 catalog_manager.cc:4841] TS ae7b39323eec4a86bc8f8f27cb547a1d (127.2.243.131:43581): tablet 6ad5449b675741bc97f221c0b0a42bc1 (table test-table [id=0cbafd08ec0f4d869e70749170474da2]) successfully deleted
W20260620 11:33:48.478819 10512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.243.129:36721 (error 108)
I20260620 11:33:48.479064 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 10739
I20260620 11:33:48.482558 11143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:48.479240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:36721 (local address 127.0.0.1:51296)
0620 11:33:48.482482 (+ 3242us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:48.482526 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.129:36721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3224,"thread_start_us":61,"threads_started":1}
W20260620 11:33:48.482672 10512 catalog_manager.cc:4584] TS 64dc8cdf08c5445d9ca51a87379c7df1 (127.2.243.129:36721): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6ad5449b675741bc97f221c0b0a42bc1: Network error: Client connection negotiation failed: client connection to 127.2.243.129:36721: connect: Connection refused (error 111)
I20260620 11:33:48.485440 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 10879
I20260620 11:33:48.490731 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 10495
2026-06-20T11:33:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3128 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:33:48 dist-test-slave-ctbw krb5kdc[11146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:33:48 dist-test-slave-ctbw krb5kdc[11146](info): set up 2 sockets
Jun 20 11:33:48 dist-test-slave-ctbw krb5kdc[11146](info): commencing operation
krb5kdc: starting...
W20260620 11:33:49.543900 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:33:49 dist-test-slave-ctbw krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:33:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:33:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:33:49.657704 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:36873
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:36479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:36873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:49.739176 11159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:49.739434 11159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:49.739475 11159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:49.740648 11159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:33:49.740717 11159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:49.740742 11159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:49.740772 11159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:33:49.740798 11159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:33:49.742363 11159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:36479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:36873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:36873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:49.742652 11159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:49.742901 11159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:49.747810 11165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:49.747813 11164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:49.747855 11159 server_base.cc:1034] running on GCE node
W20260620 11:33:49.747848 11167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:49.748335 11159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:49.748667 11159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:49.749851 11159 hybrid_clock.cc:648] HybridClock initialized: now 1781955229749836 us; error 27 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T11:33:49Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1263 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:33:49 dist-test-slave-ctbw krb5kdc[11172](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:33:49 dist-test-slave-ctbw krb5kdc[11172](info): set up 2 sockets
Jun 20 11:33:49 dist-test-slave-ctbw krb5kdc[11172](info): commencing operation
krb5kdc: starting...
W20260620 11:33:50.791797 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:33:50 dist-test-slave-ctbw krb5kdc[11172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:33:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:33:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:33:50.913197 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:33141
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:33141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:50.993283 11185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:50.993546 11185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:50.993585 11185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:50.994766 11185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:33:50.994840 11185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:50.994863 11185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:50.994881 11185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:33:50.994926 11185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:33:50.996423 11185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:33141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:33141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:50.996698 11185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:50.996930 11185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:51.001785 11190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:51.001865 11191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:51.001802 11193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:51.002035 11185 server_base.cc:1034] running on GCE node
I20260620 11:33:51.002323 11185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:51.002612 11185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:51.003801 11185 hybrid_clock.cc:648] HybridClock initialized: now 1781955231003763 us; error 50 us; skew 500 ppm
Jun 20 11:33:50 dist-test-slave-ctbw krb5kdc[11172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955230, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:51.006346 11185 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:33:51.006774 11185 webserver.cc:460] Webserver started at http://127.2.243.190:40495/ using document root <none> and password file <none>
I20260620 11:33:51.007033 11185 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:51.007107 11185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:51.007263 11185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:51.008395 11185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "212b732bbc9a4f028764eb9b206ae370"
format_stamp: "Formatted at 2026-06-20 11:33:51 on dist-test-slave-ctbw"
server_key: "07efc2dafa0e1f92635d6bbdb094865e"
server_key_iv: "0f43571ce9598422b634c9018b9c7e65"
server_key_version: "encryptionkey@0"
I20260620 11:33:51.008759 11185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "212b732bbc9a4f028764eb9b206ae370"
format_stamp: "Formatted at 2026-06-20 11:33:51 on dist-test-slave-ctbw"
server_key: "07efc2dafa0e1f92635d6bbdb094865e"
server_key_iv: "0f43571ce9598422b634c9018b9c7e65"
server_key_version: "encryptionkey@0"
I20260620 11:33:51.010098 11185 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:33:51.010846 11200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:51.011050 11185 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:51.011152 11185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "212b732bbc9a4f028764eb9b206ae370"
format_stamp: "Formatted at 2026-06-20 11:33:51 on dist-test-slave-ctbw"
server_key: "07efc2dafa0e1f92635d6bbdb094865e"
server_key_iv: "0f43571ce9598422b634c9018b9c7e65"
server_key_version: "encryptionkey@0"
I20260620 11:33:51.011240 11185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:51.036309 11185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:51.038913 11185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:51.039080 11185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:51.042840 11185 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:33141
I20260620 11:33:51.042850 11252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:33141 every 8 connection(s)
I20260620 11:33:51.043335 11185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:33:51.044173 11253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:33:51.046315 11253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370: Bootstrap starting.
I20260620 11:33:51.046934 11253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370: Neither blocks nor log segments found. Creating new log.
I20260620 11:33:51.047190 11253 log.cc:824] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:51.047825 11253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370: No bootstrap required, opened a new log
I20260620 11:33:51.048228 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 11185
I20260620 11:33:51.048313 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:33:51.048521 3022 external_mini_cluster.cc:1428] Setting key 2dc5e8f0d02435b8497741979abeac74
I20260620 11:33:51.050019 11253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212b732bbc9a4f028764eb9b206ae370" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33141 } }
I20260620 11:33:51.050194 11253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:33:51.050232 11253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b732bbc9a4f028764eb9b206ae370, State: Initialized, Role: FOLLOWER
I20260620 11:33:51.050388 11253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212b732bbc9a4f028764eb9b206ae370" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33141 } }
I20260620 11:33:51.050463 11253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:33:51.050552 11253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:33:51.050601 11253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:33:51.051730 11253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212b732bbc9a4f028764eb9b206ae370" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33141 } }
I20260620 11:33:51.051838 11253 leader_election.cc:302] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 212b732bbc9a4f028764eb9b206ae370; no voters:
I20260620 11:33:51.051983 11253 leader_election.cc:288] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:33:51.052052 11258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:33:51.052222 11253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:51.052224 11258 raft_consensus.cc:686] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [term 1 LEADER]: Becoming Leader. State: Replica: 212b732bbc9a4f028764eb9b206ae370, State: Running, Role: LEADER
I20260620 11:33:51.052438 11258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212b732bbc9a4f028764eb9b206ae370" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33141 } }
I20260620 11:33:51.052804 11258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 212b732bbc9a4f028764eb9b206ae370. Latest consensus state: current_term: 1 leader_uuid: "212b732bbc9a4f028764eb9b206ae370" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212b732bbc9a4f028764eb9b206ae370" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33141 } } }
I20260620 11:33:51.053037 11258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:51.052969 11260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "212b732bbc9a4f028764eb9b206ae370" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212b732bbc9a4f028764eb9b206ae370" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33141 } } }
I20260620 11:33:51.053436 11260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:51.053453 11267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:33:51.054171 11267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:33:51.055843 11267 catalog_manager.cc:1347] Generated new cluster ID: 32f84094cde84855b64c29753b03caf5
I20260620 11:33:51.055899 11267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:33:51 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:51.061041 11257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:51.049069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55420 (local address 127.2.243.190:33141)
0620 11:33:51.049260 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:51.049270 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:51.049302 (+ 32us) server_negotiation.cc:407] Connection header received
0620 11:33:51.049977 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:51.049986 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:51.050052 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:51.050207 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:51.051098 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.052515 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.053724 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.054186 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.054311 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:51.057073 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:51.057077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:51.057077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:51.057089 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:51.059163 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:51.059556 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:51.059560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:51.059560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:51.059693 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:51.059887 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:51.059888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:51.059888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:51.060219 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:51.060383 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:51.060468 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 11:33:51.060635 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":80,"threads_started":1}
I20260620 11:33:51.068426 11267 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:33:51.069187 11267 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:33:51.073998 11267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370: Generated new TSK 0
I20260620 11:33:51.074185 11267 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
I20260620 11:33:51.103746 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33141
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:51.185448 11279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:51.185648 11279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:51.185672 11279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:51.186848 11279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:51.186911 11279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:51.186939 11279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:33:51.188421 11279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:51.188651 11279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:51.188889 11279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:51.193730 11284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:51.193738 11287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:51.193743 11285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:51.193845 11279 server_base.cc:1034] running on GCE node
I20260620 11:33:51.194164 11279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:51.194424 11279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:51.195582 11279 hybrid_clock.cc:648] HybridClock initialized: now 1781955231195572 us; error 28 us; skew 500 ppm
Jun 20 11:33:51 dist-test-slave-ctbw krb5kdc[11172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955231, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:51.198028 11279 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:33:51.198477 11279 webserver.cc:460] Webserver started at http://127.2.243.129:36703/ using document root <none> and password file <none>
I20260620 11:33:51.198660 11279 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:51.198700 11279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:51.198784 11279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:51.199810 11279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "2acfeac5a3d2417ebdcd42edac8706df"
format_stamp: "Formatted at 2026-06-20 11:33:51 on dist-test-slave-ctbw"
server_key: "da58d15529bceec0a0a271843e40744f"
server_key_iv: "bea332b3057d489e624e9efe6e14d957"
server_key_version: "encryptionkey@0"
I20260620 11:33:51.200129 11279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "2acfeac5a3d2417ebdcd42edac8706df"
format_stamp: "Formatted at 2026-06-20 11:33:51 on dist-test-slave-ctbw"
server_key: "da58d15529bceec0a0a271843e40744f"
server_key_iv: "bea332b3057d489e624e9efe6e14d957"
server_key_version: "encryptionkey@0"
I20260620 11:33:51.201328 11279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:33:51.201980 11294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:51.202148 11279 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:51.202250 11279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "2acfeac5a3d2417ebdcd42edac8706df"
format_stamp: "Formatted at 2026-06-20 11:33:51 on dist-test-slave-ctbw"
server_key: "da58d15529bceec0a0a271843e40744f"
server_key_iv: "bea332b3057d489e624e9efe6e14d957"
server_key_version: "encryptionkey@0"
I20260620 11:33:51.202324 11279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:51.215431 11279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:51.217348 11279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:51.217517 11279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:51.217741 11279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:51.218058 11279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:51.218122 11279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:51.218155 11279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:51.218182 11279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:51.224368 11279 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:37137
I20260620 11:33:51.224429 11407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:37137 every 8 connection(s)
I20260620 11:33:51.224745 11279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:33:51.228688 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 11279
I20260620 11:33:51.228825 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:33:51.229045 3022 external_mini_cluster.cc:1428] Setting key f072fb7f0396c4ea8a885bae146a5e65
Jun 20 11:33:51 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955231, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:51.235347 11257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:51.225440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:37567 (local address 127.2.243.190:33141)
0620 11:33:51.225515 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:51.225517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:51.225712 (+ 195us) server_negotiation.cc:407] Connection header received
0620 11:33:51.226193 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:51.226194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:51.226206 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:51.226295 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:51.227345 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.227831 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.228841 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.229160 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.229243 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:51.231667 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:51.231668 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:51.231669 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:51.231677 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:51.233669 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:51.234195 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:51.234197 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:51.234197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:51.234259 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:51.234605 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:51.234605 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:51.234606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:51.234716 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:51.234774 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:51.235161 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 11:33:51.235236 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 11:33:51.235555 11410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:51.225424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.129:37567)
0620 11:33:51.225581 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:51.225622 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:51.226098 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:51.226345 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:51.226349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:51.226977 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:51.227168 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:51.227174 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.227931 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.227932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:51.228702 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:51.228704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.229263 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.229264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:51.229400 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:51.229941 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:51.229949 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:51.231573 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:51.233745 (+ 2172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:51.233751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:51.233753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:51.234086 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:51.234374 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:51.234375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:51.234375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:51.234525 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:51.234788 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:51.234794 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:51.235004 (+ 210us) client_negotiation.cc:769] Sending connection context
0620 11:33:51.235136 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 11:33:51.235305 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
I20260620 11:33:51.236284 11408 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33141
I20260620 11:33:51.236385 11408 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:51.236581 11408 heartbeater.cc:511] Master 127.2.243.190:33141 requested a full tablet report, sending...
I20260620 11:33:51.237200 11217 ts_manager.cc:194] Registered new tserver with Master: 2acfeac5a3d2417ebdcd42edac8706df (127.2.243.129:37137)
WARNING: no policy specified for kudu/127.2.243.130@KRBTEST.COM; defaulting to no policy
I20260620 11:33:51.238113 11217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:37567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
I20260620 11:33:51.254328 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33141
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:51.334163 11413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:51.334349 11413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:51.334371 11413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:51.335453 11413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:51.335485 11413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:51.335511 11413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:33:51.336848 11413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:51.337023 11413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:51.337191 11413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:51.342293 11419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:51.342352 11421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:51.342293 11418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:51.342593 11413 server_base.cc:1034] running on GCE node
I20260620 11:33:51.342803 11413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:51.343056 11413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:51.344197 11413 hybrid_clock.cc:648] HybridClock initialized: now 1781955231344176 us; error 39 us; skew 500 ppm
Jun 20 11:33:51 dist-test-slave-ctbw krb5kdc[11172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955231, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:51.346535 11413 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:33:51.347074 11413 webserver.cc:460] Webserver started at http://127.2.243.130:37041/ using document root <none> and password file <none>
I20260620 11:33:51.347317 11413 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:51.347384 11413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:51.347518 11413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:51.348567 11413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "2a76237aa4d54b168878d8f95e7e9419"
format_stamp: "Formatted at 2026-06-20 11:33:51 on dist-test-slave-ctbw"
server_key: "e11dc1d5920dfc0d15a2fa87d4c2a55b"
server_key_iv: "31523a45202b8761a8844d929ea9bde5"
server_key_version: "encryptionkey@0"
I20260620 11:33:51.348932 11413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "2a76237aa4d54b168878d8f95e7e9419"
format_stamp: "Formatted at 2026-06-20 11:33:51 on dist-test-slave-ctbw"
server_key: "e11dc1d5920dfc0d15a2fa87d4c2a55b"
server_key_iv: "31523a45202b8761a8844d929ea9bde5"
server_key_version: "encryptionkey@0"
I20260620 11:33:51.350193 11413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:33:51.350929 11428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:51.351095 11413 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:51.351190 11413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "2a76237aa4d54b168878d8f95e7e9419"
format_stamp: "Formatted at 2026-06-20 11:33:51 on dist-test-slave-ctbw"
server_key: "e11dc1d5920dfc0d15a2fa87d4c2a55b"
server_key_iv: "31523a45202b8761a8844d929ea9bde5"
server_key_version: "encryptionkey@0"
I20260620 11:33:51.351275 11413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:51.372519 11413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:51.374598 11413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:51.374769 11413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:51.375082 11413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:51.375412 11413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:51.375473 11413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:51.375515 11413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:51.375540 11413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:51.381273 11413 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:42625
I20260620 11:33:51.381325 11541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:42625 every 8 connection(s)
I20260620 11:33:51.381618 11413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
Jun 20 11:33:51 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955231, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:51.389015 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 11413
I20260620 11:33:51.389124 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:33:51.389343 3022 external_mini_cluster.cc:1428] Setting key cb37ebffb827d6273f88d0adfee88f71
I20260620 11:33:51.391724 11257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:51.382275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:50717 (local address 127.2.243.190:33141)
0620 11:33:51.382322 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:51.382324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:51.382544 (+ 220us) server_negotiation.cc:407] Connection header received
0620 11:33:51.383418 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:51.383418 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:51.383431 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:51.383484 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:51.384434 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.384927 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.385925 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.386122 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.386208 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:51.388292 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:51.388293 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:51.388293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:51.388301 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:51.389893 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:51.390549 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:51.390550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:51.390551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:51.390604 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:51.390969 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:51.390969 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:51.390970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:51.391077 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:51.391165 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:51.391509 (+ 344us) server_negotiation.cc:299] Negotiation successful
0620 11:33:51.391597 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:33:51.391927 11544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:51.382238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.130:50717)
0620 11:33:51.382423 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:51.382458 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:51.383247 (+ 789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:51.383557 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:51.383561 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:51.384136 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:51.384320 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:51.384324 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.385030 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.385031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:51.385833 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:51.385835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.386214 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.386216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:51.386350 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:51.386879 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:51.386887 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:51.388180 (+ 1293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:51.390006 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:51.390012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:51.390014 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:51.390466 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:51.390679 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:51.390680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:51.390680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:51.390896 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:51.391169 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:51.391175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:51.391353 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 11:33:51.391476 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 11:33:51.391664 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":96,"threads_started":1}
I20260620 11:33:51.392369 11542 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33141
I20260620 11:33:51.392477 11542 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:51.392668 11542 heartbeater.cc:511] Master 127.2.243.190:33141 requested a full tablet report, sending...
I20260620 11:33:51.393071 11217 ts_manager.cc:194] Registered new tserver with Master: 2a76237aa4d54b168878d8f95e7e9419 (127.2.243.130:42625)
I20260620 11:33:51.393436 11217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:50717
WARNING: no policy specified for kudu/127.2.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
I20260620 11:33:51.415820 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33141
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:51.496697 11547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:51.496893 11547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:51.496915 11547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:51.498085 11547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:51.498129 11547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:51.498157 11547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:33:51.499651 11547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:51.499890 11547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:51.500121 11547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:51.505270 11552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:51.505242 11553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:51.505527 11547 server_base.cc:1034] running on GCE node
W20260620 11:33:51.505548 11555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:51.505760 11547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:51.506013 11547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:51.507176 11547 hybrid_clock.cc:648] HybridClock initialized: now 1781955231507156 us; error 49 us; skew 500 ppm
Jun 20 11:33:51 dist-test-slave-ctbw krb5kdc[11172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955231, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:51.509697 11547 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:33:51.510185 11547 webserver.cc:460] Webserver started at http://127.2.243.131:37031/ using document root <none> and password file <none>
I20260620 11:33:51.510419 11547 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:51.510488 11547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:51.510638 11547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:33:51.511785 11547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "09e59f188e1f4772b66fc3e2e15208b9"
format_stamp: "Formatted at 2026-06-20 11:33:51 on dist-test-slave-ctbw"
server_key: "79500fd850aa686a0d662e9c420374d5"
server_key_iv: "dc84ec905627ff1c458f7b49a04c606b"
server_key_version: "encryptionkey@0"
I20260620 11:33:51.512142 11547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "09e59f188e1f4772b66fc3e2e15208b9"
format_stamp: "Formatted at 2026-06-20 11:33:51 on dist-test-slave-ctbw"
server_key: "79500fd850aa686a0d662e9c420374d5"
server_key_iv: "dc84ec905627ff1c458f7b49a04c606b"
server_key_version: "encryptionkey@0"
I20260620 11:33:51.513545 11547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:33:51.514307 11562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:51.514492 11547 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:51.514590 11547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "09e59f188e1f4772b66fc3e2e15208b9"
format_stamp: "Formatted at 2026-06-20 11:33:51 on dist-test-slave-ctbw"
server_key: "79500fd850aa686a0d662e9c420374d5"
server_key_iv: "dc84ec905627ff1c458f7b49a04c606b"
server_key_version: "encryptionkey@0"
I20260620 11:33:51.514675 11547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:51.529969 11547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:51.532001 11547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:51.532161 11547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:51.532387 11547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:51.532697 11547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:51.532749 11547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:51.532789 11547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:51.532816 11547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:51.538478 11547 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:35353
I20260620 11:33:51.538612 11675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:35353 every 8 connection(s)
I20260620 11:33:51.538903 11547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:33:51.540700 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 11547
I20260620 11:33:51.540827 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:33:51.541025 3022 external_mini_cluster.cc:1428] Setting key 537a25f27a804240274c04b668295eff
Jun 20 11:33:51 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955231, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:33:51.548241 11257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:51.539637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:44017 (local address 127.2.243.190:33141)
0620 11:33:51.539713 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:51.539716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:51.539862 (+ 146us) server_negotiation.cc:407] Connection header received
0620 11:33:51.540337 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:51.540338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:51.540347 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:51.540398 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:51.541506 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.542150 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.543218 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.543395 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.543444 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:51.545583 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:51.545584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:51.545585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:51.545591 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:51.547044 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:51.547451 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:51.547451 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:51.547451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:51.547499 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:51.547767 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:51.547768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:51.547768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:51.547855 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:51.547900 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:51.548128 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 11:33:51.548181 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 11:33:51.548542 11678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:51.539584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.131:44017)
0620 11:33:51.539754 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:51.539790 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:51.540252 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:51.540453 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:51.540457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:51.541133 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:51.541407 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:51.541417 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.542229 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.542230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:51.543137 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:51.543140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.543463 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.543464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:51.543566 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:51.544096 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:51.544104 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:51.545498 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:33:51.547116 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:51.547122 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:51.547123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:51.547395 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:51.547554 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:33:51.547555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:33:51.547555 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:33:51.547692 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:33:51.547907 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:33:51.547912 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:33:51.548021 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 11:33:51.548115 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 11:33:51.548230 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260620 11:33:51.549100 11676 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33141
I20260620 11:33:51.549201 11676 heartbeater.cc:464] Registering TS with master...
I20260620 11:33:51.549388 11676 heartbeater.cc:511] Master 127.2.243.190:33141 requested a full tablet report, sending...
I20260620 11:33:51.549744 11217 ts_manager.cc:194] Registered new tserver with Master: 09e59f188e1f4772b66fc3e2e15208b9 (127.2.243.131:35353)
I20260620 11:33:51.550130 11217 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:44017
I20260620 11:33:51.553051 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.2.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.129.keytab.
Entry for principal oryx/127.2.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.129.keytab.
WARNING: no policy specified for oryx/127.2.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.130.keytab.
Entry for principal oryx/127.2.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.130.keytab.
WARNING: no policy specified for oryx/127.2.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.2.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.2.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.131.keytab.
Entry for principal oryx/127.2.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.131.keytab.
I20260620 11:33:51.619495 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 11279
I20260620 11:33:51.624006 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 11413
I20260620 11:33:51.628139 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 11547
I20260620 11:33:51.632328 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 11185
I20260620 11:33:51.637239 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:33141
--webserver_interface=127.2.243.190
--webserver_port=40495
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:33141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:33:51.717804 11685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:51.718040 11685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:51.718076 11685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:51.719293 11685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:33:51.719362 11685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:51.719389 11685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:51.719422 11685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:33:51.719445 11685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:33:51.720904 11685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:33141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:33141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=40495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:51.721170 11685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:51.721400 11685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:51.726137 11691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:51.726176 11693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:51.726277 11690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:51.726634 11685 server_base.cc:1034] running on GCE node
I20260620 11:33:51.726824 11685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:51.727166 11685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:51.728317 11685 hybrid_clock.cc:648] HybridClock initialized: now 1781955231728303 us; error 30 us; skew 500 ppm
Jun 20 11:33:51 dist-test-slave-ctbw krb5kdc[11172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955231, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:51.730722 11685 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:33:51.731208 11685 webserver.cc:460] Webserver started at http://127.2.243.190:40495/ using document root <none> and password file <none>
I20260620 11:33:51.731441 11685 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:51.731523 11685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:51.732874 11685 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:51.733527 11700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:51.733696 11685 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:51.733795 11685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "212b732bbc9a4f028764eb9b206ae370"
format_stamp: "Formatted at 2026-06-20 11:33:51 on dist-test-slave-ctbw"
server_key: "07efc2dafa0e1f92635d6bbdb094865e"
server_key_iv: "0f43571ce9598422b634c9018b9c7e65"
server_key_version: "encryptionkey@0"
I20260620 11:33:51.734156 11685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:51.746304 11685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:51.748219 11685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:51.748385 11685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:51.752379 11685 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:33141
I20260620 11:33:51.752414 11752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:33141 every 8 connection(s)
I20260620 11:33:51.752830 11685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:33:51.754276 11753 sys_catalog.cc:263] Verifying existing consensus state
I20260620 11:33:51.754987 11753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370: Bootstrap starting.
I20260620 11:33:51.761826 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 11685
I20260620 11:33:51.762188 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:37137
--local_ip_for_outbound_sockets=127.2.243.129
--tserver_master_addrs=127.2.243.190:33141
--webserver_port=36703
--webserver_interface=127.2.243.129
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.243.129
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:33:51.763195 11753 log.cc:824] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370: Log is configured to *not* fsync() on all Append() calls
I20260620 11:33:51.765029 11753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:33:51.765308 11753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370: Bootstrap complete.
I20260620 11:33:51.767700 11753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212b732bbc9a4f028764eb9b206ae370" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33141 } }
I20260620 11:33:51.767946 11753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212b732bbc9a4f028764eb9b206ae370, State: Initialized, Role: FOLLOWER
I20260620 11:33:51.768159 11753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212b732bbc9a4f028764eb9b206ae370" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33141 } }
I20260620 11:33:51.768256 11753 raft_consensus.cc:388] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:33:51.768309 11753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:33:51.768357 11753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [term 1 FOLLOWER]: Advancing to term 2
I20260620 11:33:51.769335 11753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212b732bbc9a4f028764eb9b206ae370" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33141 } }
I20260620 11:33:51.769464 11753 leader_election.cc:302] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 212b732bbc9a4f028764eb9b206ae370; no voters:
I20260620 11:33:51.769665 11753 leader_election.cc:288] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 11:33:51.769697 11757 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 11:33:51.769835 11757 raft_consensus.cc:686] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [term 2 LEADER]: Becoming Leader. State: Replica: 212b732bbc9a4f028764eb9b206ae370, State: Running, Role: LEADER
I20260620 11:33:51.769940 11753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:33:51.769922 11757 consensus_queue.cc:237] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212b732bbc9a4f028764eb9b206ae370" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33141 } }
I20260620 11:33:51.770232 11757 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 212b732bbc9a4f028764eb9b206ae370. Latest consensus state: current_term: 2 leader_uuid: "212b732bbc9a4f028764eb9b206ae370" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212b732bbc9a4f028764eb9b206ae370" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33141 } } }
I20260620 11:33:51.770300 11757 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:51.770454 11758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "212b732bbc9a4f028764eb9b206ae370" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "212b732bbc9a4f028764eb9b206ae370" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33141 } } }
I20260620 11:33:51.770612 11758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370 [sys.catalog]: This master's current role is: LEADER
I20260620 11:33:51.770539 11762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:33:51.771330 11762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:33:51.771930 11762 catalog_manager.cc:1259] Loaded cluster ID: 32f84094cde84855b64c29753b03caf5
I20260620 11:33:51.771981 11762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:33:51.773280 11762 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:33:51.773458 11762 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 212b732bbc9a4f028764eb9b206ae370: Loaded TSK: 0
I20260620 11:33:51.773672 11762 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 11:33:51.845759 11756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:51.845963 11756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:51.845988 11756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:51.847220 11756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:51.847260 11756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:51.847286 11756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:33:51.848685 11756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.129.keytab
--principal=oryx/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:37137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=36703
--tserver_master_addrs=127.2.243.190:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:51.848924 11756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:51.849169 11756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:51.854017 11779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:51.854127 11780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:51.853999 11782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:51.854715 11756 server_base.cc:1034] running on GCE node
I20260620 11:33:51.854997 11756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:51.855233 11756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:51.856408 11756 hybrid_clock.cc:648] HybridClock initialized: now 1781955231856383 us; error 46 us; skew 500 ppm
Jun 20 11:33:51 dist-test-slave-ctbw krb5kdc[11172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955231, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:51.858994 11756 init.cc:377] Logged in from keytab as oryx/127.2.243.129@KRBTEST.COM (short username oryx)
I20260620 11:33:51.859478 11756 webserver.cc:460] Webserver started at http://127.2.243.129:36703/ using document root <none> and password file <none>
I20260620 11:33:51.859743 11756 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:51.859802 11756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:51.861191 11756 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:51.861819 11789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:51.861974 11756 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:51.862056 11756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "2acfeac5a3d2417ebdcd42edac8706df"
format_stamp: "Formatted at 2026-06-20 11:33:51 on dist-test-slave-ctbw"
server_key: "da58d15529bceec0a0a271843e40744f"
server_key_iv: "bea332b3057d489e624e9efe6e14d957"
server_key_version: "encryptionkey@0"
I20260620 11:33:51.862385 11756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:51.889643 11756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:51.892654 11756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:51.892830 11756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:51.893106 11756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:51.893481 11756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:51.893530 11756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:51.893612 11756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:51.893638 11756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:51.900255 11756 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:37137
I20260620 11:33:51.900300 11902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:37137 every 8 connection(s)
I20260620 11:33:51.900651 11756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
Jun 20 11:33:51 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:51.907703 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 11756
I20260620 11:33:51.908039 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:42625
--local_ip_for_outbound_sockets=127.2.243.130
--tserver_master_addrs=127.2.243.190:33141
--webserver_port=37041
--webserver_interface=127.2.243.130
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.243.130
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 11:33:51 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:51.909543 11905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:51.901376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.129:59929)
0620 11:33:51.901581 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:51.901629 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:51.902143 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:51.902616 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:51.902620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:51.903265 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:51.903456 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:51.903461 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.904560 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.904561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:51.905341 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:51.905343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.905966 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.905967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:51.906105 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:51.906627 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:51.906635 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:51.909291 (+ 2656us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":101,"threads_started":1}
W20260620 11:33:51.909789 11903 heartbeater.cc:644] Failed to heartbeat to 127.2.243.190:33141 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.243.190:33141: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
W20260620 11:33:51.909811 11906 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:51.901438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:59929 (local address 127.2.243.190:33141)
0620 11:33:51.901637 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:51.901646 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:51.901728 (+ 82us) server_negotiation.cc:407] Connection header received
0620 11:33:51.902388 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:51.902394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:51.902436 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:51.902579 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:51.903573 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.904453 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.905446 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.905862 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.905991 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:51.909601 (+ 3610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.129:59929: BlockingRecv error: recv got EOF from 127.2.243.129:59929 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":106,"threads_started":1}
Jun 20 11:33:51 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:51 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:51.915191 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:51.909995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.129:60413)
0620 11:33:51.910135 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:51.910146 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:51.910207 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:51.910395 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:51.910396 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:51.910519 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:51.910560 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:51.910561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.911534 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.911535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:51.912056 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:51.912058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.912631 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.912632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:51.912681 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:51.913225 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:51.913230 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:51.915137 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
W20260620 11:33:51.915361 11906 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:51.910060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:60413 (local address 127.2.243.190:33141)
0620 11:33:51.910121 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:51.910123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:51.910260 (+ 137us) server_negotiation.cc:407] Connection header received
0620 11:33:51.910286 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:51.910287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:51.910303 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:51.910400 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:51.910634 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.911452 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.912142 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.912508 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.912598 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:51.915263 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.129:60413: BlockingRecv error: recv got EOF from 127.2.243.129:60413 (error 108)
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":42}
Jun 20 11:33:51 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:51 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:51.919853 11908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:51.915567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.129:35279)
0620 11:33:51.915617 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:51.915627 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:51.915690 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:51.915806 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:51.915807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:51.915926 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:51.915951 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:51.915952 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.916572 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.916573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:51.917056 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:51.917057 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.917406 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.917407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:51.917471 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:51.918003 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:51.918012 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:51.919777 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 11:33:51.919905 11906 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:51.915638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:35279 (local address 127.2.243.190:33141)
0620 11:33:51.915701 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:51.915703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:51.915713 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:33:51.915736 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:51.915737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:51.915747 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:51.915803 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:51.916038 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.916491 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.917146 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:51.917326 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:51.917405 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:51.919842 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.129:35279: BlockingRecv error: recv got EOF from 127.2.243.129:35279 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 11:33:51.919991 11903 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 11:33:51.991665 11907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:51.991902 11907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:51.991940 11907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:51.993160 11907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:51.993214 11907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:51.993258 11907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:33:51.994692 11907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.130.keytab
--principal=oryx/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:42625
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=37041
--tserver_master_addrs=127.2.243.190:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:51.994977 11907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:51.995203 11907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:51.999986 11916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:52.000036 11914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:52.000005 11913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:52.000451 11907 server_base.cc:1034] running on GCE node
I20260620 11:33:52.000645 11907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:52.000828 11907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:52.001991 11907 hybrid_clock.cc:648] HybridClock initialized: now 1781955232001975 us; error 27 us; skew 500 ppm
Jun 20 11:33:51 dist-test-slave-ctbw krb5kdc[11172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955231, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:52.004443 11907 init.cc:377] Logged in from keytab as oryx/127.2.243.130@KRBTEST.COM (short username oryx)
I20260620 11:33:52.004866 11907 webserver.cc:460] Webserver started at http://127.2.243.130:37041/ using document root <none> and password file <none>
I20260620 11:33:52.005095 11907 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:52.005177 11907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:52.006464 11907 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:52.007110 11923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:52.007292 11907 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:52.007391 11907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "2a76237aa4d54b168878d8f95e7e9419"
format_stamp: "Formatted at 2026-06-20 11:33:51 on dist-test-slave-ctbw"
server_key: "e11dc1d5920dfc0d15a2fa87d4c2a55b"
server_key_iv: "31523a45202b8761a8844d929ea9bde5"
server_key_version: "encryptionkey@0"
I20260620 11:33:52.007740 11907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:52.016166 11907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:52.018275 11907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:52.018515 11907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:52.018759 11907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:52.019124 11907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:52.019191 11907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:52.019228 11907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:52.019259 11907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:52.025431 11907 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:42625
I20260620 11:33:52.025518 12036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:42625 every 8 connection(s)
I20260620 11:33:52.025869 11907 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
Jun 20 11:33:52 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:52 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
W20260620 11:33:52.033047 11906 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:52.026640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:33245 (local address 127.2.243.190:33141)
0620 11:33:52.026706 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:52.026708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:52.026924 (+ 216us) server_negotiation.cc:407] Connection header received
0620 11:33:52.027454 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:52.027455 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:52.027469 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:52.027530 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:52.028404 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.028895 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.029868 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.030059 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.030122 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:52.033002 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.130:33245: BlockingRecv error: recv got EOF from 127.2.243.130:33245 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 11:33:52.033051 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:52.026628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.130:33245)
0620 11:33:52.026792 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:52.026833 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:52.027331 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:52.027582 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:52.027585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:52.028147 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:52.028330 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:52.028334 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.028988 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.028989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:52.029777 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:52.029779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.030134 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.030135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:52.030230 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:52.030739 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:52.030747 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:52.032905 (+ 2158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
W20260620 11:33:52.033351 12037 heartbeater.cc:644] Failed to heartbeat to 127.2.243.190:33141 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.243.190:33141: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
I20260620 11:33:52.033718 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 11907
I20260620 11:33:52.034050 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:35353
--local_ip_for_outbound_sockets=127.2.243.131
--tserver_master_addrs=127.2.243.190:33141
--webserver_port=37031
--webserver_interface=127.2.243.131
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:44399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.2.243.131
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 11:33:52 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:52 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:52.038461 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:52.033638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.130:59463)
0620 11:33:52.033680 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:52.033691 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:52.033726 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:52.033885 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:52.033886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:52.034011 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:52.034053 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:52.034054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.034688 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.034689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:52.035475 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:52.035477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.035802 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.035803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:52.035871 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:52.036282 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:52.036287 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:52.038352 (+ 2065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 11:33:52.038475 11906 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:52.033681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:59463 (local address 127.2.243.190:33141)
0620 11:33:52.033719 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:52.033721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:52.033730 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:33:52.033802 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:52.033803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:52.033816 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:52.033879 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:52.034139 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.034609 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.035555 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.035731 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.035786 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:52.038431 (+ 2645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.130:59463: BlockingRecv error: recv got EOF from 127.2.243.130:59463 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Jun 20 11:33:52 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:52 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:52.046267 12039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:52.038834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.130:55875)
0620 11:33:52.038900 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:52.038911 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:52.038956 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:52.039084 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:52.039085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:52.039188 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:52.039213 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:52.039214 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.040125 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.040125 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:52.040590 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:52.040591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.041073 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.041074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:52.041108 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:52.044580 (+ 3472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:52.044584 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:52.046217 (+ 1633us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 11:33:52.046381 11906 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:52.038914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:55875 (local address 127.2.243.190:33141)
0620 11:33:52.038958 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:52.038960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:52.038985 (+ 25us) server_negotiation.cc:407] Connection header received
0620 11:33:52.039010 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:52.039011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:52.039021 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:52.039083 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:52.039278 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.040047 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.040664 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.040977 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.041079 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:52.046319 (+ 5240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.130:55875: BlockingRecv error: recv got EOF from 127.2.243.130:55875 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 11:33:52.046427 12037 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 11:33:52.115650 12040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:33:52.115847 12040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:33:52.115870 12040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:33:52.116962 12040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:33:52.116997 12040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:33:52.117022 12040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:33:52.118446 12040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:44399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/krb5kdc/oryx_127.2.243.131.keytab
--principal=oryx/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:35353
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=37031
--tserver_master_addrs=127.2.243.190:33141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:33:52.118618 12040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:33:52.118786 12040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:33:52.123528 12045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:52.123476 12046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:33:52.123538 12048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:33:52.123783 12040 server_base.cc:1034] running on GCE node
I20260620 11:33:52.123983 12040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:33:52.124178 12040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:33:52.125319 12040 hybrid_clock.cc:648] HybridClock initialized: now 1781955232125296 us; error 39 us; skew 500 ppm
Jun 20 11:33:52 dist-test-slave-ctbw krb5kdc[11172](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955232, etypes {rep=17 tkt=17 ses=17}, oryx/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:33:52.127764 12040 init.cc:377] Logged in from keytab as oryx/127.2.243.131@KRBTEST.COM (short username oryx)
I20260620 11:33:52.128170 12040 webserver.cc:460] Webserver started at http://127.2.243.131:37031/ using document root <none> and password file <none>
I20260620 11:33:52.128391 12040 fs_manager.cc:359] Metadata directory not provided
I20260620 11:33:52.128473 12040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:33:52.129848 12040 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:33:52.130436 12055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:52.130632 12040 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:33:52.130734 12040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "09e59f188e1f4772b66fc3e2e15208b9"
format_stamp: "Formatted at 2026-06-20 11:33:51 on dist-test-slave-ctbw"
server_key: "79500fd850aa686a0d662e9c420374d5"
server_key_iv: "dc84ec905627ff1c458f7b49a04c606b"
server_key_version: "encryptionkey@0"
I20260620 11:33:52.131115 12040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:33:52.147049 12040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:33:52.148974 12040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:33:52.149152 12040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:33:52.149381 12040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:33:52.149708 12040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:33:52.149763 12040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:52.149804 12040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:33:52.149832 12040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:33:52.155776 12040 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:35353
I20260620 11:33:52.155817 12168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:35353 every 8 connection(s)
I20260620 11:33:52.156157 12040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:33:52.159422 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 12040
Jun 20 11:33:52 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:52 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:52.163829 12171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:52.156796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.131:37021)
0620 11:33:52.156984 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:52.157019 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:52.157508 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:52.157732 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:52.157736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:52.158308 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:52.158503 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:52.158508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.159219 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.159219 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:52.160239 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:52.160242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.160564 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.160565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:52.160714 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:52.161259 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:52.161267 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:52.163684 (+ 2417us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":88,"threads_started":1}
W20260620 11:33:52.163862 11906 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:52.156797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:37021 (local address 127.2.243.190:33141)
0620 11:33:52.156866 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:52.156868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:52.157089 (+ 221us) server_negotiation.cc:407] Connection header received
0620 11:33:52.157594 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:52.157595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:52.157605 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:52.157660 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:52.158578 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.159142 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.160312 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.160486 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.160535 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:52.163790 (+ 3255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.131:37021: BlockingRecv error: recv got EOF from 127.2.243.131:37021 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 11:33:52.164076 12169 heartbeater.cc:644] Failed to heartbeat to 127.2.243.190:33141 (0 consecutive failures): Not authorized: Failed to ping master at 127.2.243.190:33141: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
I20260620 11:33:52.167830 12173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:52.159957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55436 (local address 127.2.243.190:33141)
0620 11:33:52.160066 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:52.160068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:52.160077 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:33:52.160125 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:52.160125 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:52.160134 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:52.160186 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:52.160884 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.161473 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.162543 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.162725 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.162806 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:52.164220 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:33:52.164224 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:33:52.164225 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:33:52.164236 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:33:52.166127 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:52.166553 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:52.166556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:52.166556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:52.166697 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:33:52.166944 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:33:52.166945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:33:52.166946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:33:52.167285 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:33:52.167495 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 11:33:52.167573 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 11:33:52.167700 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":74,"threads_started":1}
Jun 20 11:33:52 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:52 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:52.169301 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:52.164257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.131:56391)
0620 11:33:52.164379 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:52.164389 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:52.164445 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:52.164576 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:52.164577 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:52.164745 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:52.164794 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:52.164796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.165701 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.165702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:52.166267 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:52.166268 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.166719 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.166720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:52.166800 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:52.167238 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:52.167250 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:52.169243 (+ 1993us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
W20260620 11:33:52.169359 11906 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:52.164295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:56391 (local address 127.2.243.190:33141)
0620 11:33:52.164330 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:52.164332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:52.164473 (+ 141us) server_negotiation.cc:407] Connection header received
0620 11:33:52.164501 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:52.164502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:52.164511 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:52.164568 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:52.164875 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.165627 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.166345 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.166611 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.166661 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:52.169321 (+ 2660us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.131:56391: BlockingRecv error: recv got EOF from 127.2.243.131:56391 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Jun 20 11:33:52 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:52 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:52.173422 12174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:52.169684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.131:42631)
0620 11:33:52.169729 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:52.169741 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:52.169786 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:52.169912 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:52.169912 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:52.170025 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:52.170051 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:52.170052 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.170594 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.170594 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:52.171257 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:52.171258 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.171570 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.171571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:52.171613 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:52.171916 (+ 303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:52.171920 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:52.173382 (+ 1462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 11:33:52.173549 11906 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:52.169725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:42631 (local address 127.2.243.190:33141)
0620 11:33:52.169762 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:52.169763 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:52.169773 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:33:52.169850 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:52.169851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:52.169858 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:52.169909 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:52.170106 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.170528 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.171326 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.171490 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.171557 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:52.173465 (+ 1908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.131:42631: BlockingRecv error: recv got EOF from 127.2.243.131:42631 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 11:33:52.173565 12169 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 11:33:52 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:52 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:52.925002 12175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:52.920316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.129:47351)
0620 11:33:52.920458 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:52.920470 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:52.920558 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:52.920716 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:52.920717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:52.920926 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:52.920987 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:52.920989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.921592 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.921593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:52.922088 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:52.922089 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.922397 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.922398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:52.922438 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:52.922943 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:52.922947 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:52.924904 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
W20260620 11:33:52.925266 12176 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:52.920397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:47351 (local address 127.2.243.190:33141)
0620 11:33:52.920522 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:52.920524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:52.920531 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:33:52.920620 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:52.920621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:52.920630 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:52.920689 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:52.921056 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.921526 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.922156 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:52.922331 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:52.922379 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:52.925050 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.129:47351: BlockingRecv error: recv got EOF from 127.2.243.129:47351 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":64,"threads_started":1}
Jun 20 11:33:53 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:53 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:53.051326 12177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:53.046807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.130:56483)
0620 11:33:53.046969 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:53.046981 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:53.047050 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:53.047175 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:53.047175 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:53.047343 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:53.047389 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:53.047391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:53.048029 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:53.048031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:53.048644 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:53.048646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:53.048945 (+ 299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:53.048946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:53.048985 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:53.049413 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:53.049417 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:53.051254 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":101,"threads_started":1}
W20260620 11:33:53.051445 12176 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:53.046884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:56483 (local address 127.2.243.190:33141)
0620 11:33:53.046948 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:53.046951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:53.047062 (+ 111us) server_negotiation.cc:407] Connection header received
0620 11:33:53.047115 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:53.047116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:53.047127 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:53.047170 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:53.047447 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:53.047945 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:53.048716 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:53.048887 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:53.048933 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:53.051346 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.130:56483: BlockingRecv error: recv got EOF from 127.2.243.130:56483 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 11:33:53 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:53 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:53.179150 12178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:53.173978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.131:36545)
0620 11:33:53.174175 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:53.174188 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:53.174301 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:53.174452 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:53.174453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:53.174642 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:53.174687 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:53.174689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:53.175450 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:53.175452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:53.175986 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:53.175987 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:53.176458 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:53.176459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:53.176536 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:53.177126 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:53.177132 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:53.179082 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":100,"threads_started":1}
W20260620 11:33:53.179246 12176 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:53.174094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:36545 (local address 127.2.243.190:33141)
0620 11:33:53.174179 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:53.174182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:53.174275 (+ 93us) server_negotiation.cc:407] Connection header received
0620 11:33:53.174380 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:53.174381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:53.174393 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:53.174448 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:53.174772 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:53.175355 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:53.176091 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:53.176331 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:53.176404 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:53.179179 (+ 2775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.131:36545: BlockingRecv error: recv got EOF from 127.2.243.131:36545 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 11:33:53 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:53 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:53.930565 12179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:53.925651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.129:41021)
0620 11:33:53.925802 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:53.925812 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:53.925863 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:53.926015 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:53.926016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:53.926178 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:53.926226 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:53.926227 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:53.926954 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:53.926955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:53.927545 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:53.927547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:53.927898 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:53.927899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:53.927939 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:53.928488 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:53.928493 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:53.930504 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
W20260620 11:33:53.930740 12180 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:53.925720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:41021 (local address 127.2.243.190:33141)
0620 11:33:53.925855 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:53.925856 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:53.925866 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:33:53.925924 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:53.925925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:53.925935 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:53.925999 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:53.926292 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:53.926829 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:53.927615 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:53.927833 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:53.927884 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:53.930623 (+ 2739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.129:41021: BlockingRecv error: recv got EOF from 127.2.243.129:41021 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
Jun 20 11:33:54 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:54 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:54.056100 12181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:54.051759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.130:45827)
0620 11:33:54.051890 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:54.051901 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:54.051955 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:54.052079 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:54.052080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:54.052254 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:54.052303 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:54.052304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:54.052941 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:54.052942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:54.053492 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:54.053493 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:54.053857 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:54.053857 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:54.053896 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:54.054330 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:54.054335 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:54.056041 (+ 1706us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":77,"threads_started":1}
W20260620 11:33:54.056205 12180 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:54.051846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:45827 (local address 127.2.243.190:33141)
0620 11:33:54.051909 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:54.051915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:54.051942 (+ 27us) server_negotiation.cc:407] Connection header received
0620 11:33:54.052010 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:54.052010 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:54.052021 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:54.052062 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:54.052369 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:54.052873 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:54.053576 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:54.053789 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:54.053846 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:54.056138 (+ 2292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.130:45827: BlockingRecv error: recv got EOF from 127.2.243.130:45827 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 11:33:54 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:54 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:54.184191 12182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:54.179582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.131:35643)
0620 11:33:54.179716 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:54.179726 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:54.179831 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:54.179989 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:54.179990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:54.180160 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:54.180203 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:54.180205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:54.180877 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:54.180879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:54.181400 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:54.181402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:54.181780 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:54.181781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:54.181844 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:54.182300 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:54.182305 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:54.184096 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
W20260620 11:33:54.184250 12180 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:54.179756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:35643 (local address 127.2.243.190:33141)
0620 11:33:54.179825 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:54.179827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:54.179836 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:33:54.179883 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:54.179884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:54.179899 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:54.179967 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:54.180303 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:54.180786 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:54.181482 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:54.181670 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:54.181767 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:54.184177 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.131:35643: BlockingRecv error: recv got EOF from 127.2.243.131:35643 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 11:33:54 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:54 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:54.935812 12183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:54.931070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.129:46631)
0620 11:33:54.931188 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:54.931204 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:54.931269 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:54.931412 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:54.931413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:54.931573 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:54.931625 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:54.931649 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:54.932285 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:54.932286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:54.932882 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:54.932884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:54.933243 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:54.933244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:54.933296 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:54.933848 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:54.933853 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:54.935755 (+ 1902us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
W20260620 11:33:54.935940 12184 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:54.931108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:46631 (local address 127.2.243.190:33141)
0620 11:33:54.931219 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:54.931221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:54.931252 (+ 31us) server_negotiation.cc:407] Connection header received
0620 11:33:54.931323 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:54.931324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:54.931348 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:54.931406 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:54.931723 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:54.932222 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:54.932956 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:54.933136 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:54.933225 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:54.935853 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.129:46631: BlockingRecv error: recv got EOF from 127.2.243.129:46631 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":66,"threads_started":1}
Jun 20 11:33:55 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:55 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:55.061803 12185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:55.056735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.130:40551)
0620 11:33:55.056898 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:55.056911 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:55.056988 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:55.057141 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:55.057142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:55.057335 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:55.057389 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:55.057391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:55.058130 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:55.058131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:55.058812 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:55.058816 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:55.059366 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:55.059367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:55.059434 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:55.059956 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:55.059961 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:55.061745 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":113,"threads_started":1}
W20260620 11:33:55.062024 12184 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:55.056826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:40551 (local address 127.2.243.190:33141)
0620 11:33:55.056874 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:55.056876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:55.056986 (+ 110us) server_negotiation.cc:407] Connection header received
0620 11:33:55.057050 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:55.057051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:55.057062 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:55.057118 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:55.057520 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:55.058035 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:55.058996 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:55.059284 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:55.059365 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:55.061943 (+ 2578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.130:40551: BlockingRecv error: recv got EOF from 127.2.243.130:40551 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 20 11:33:55 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:55 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:55.189633 12186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:55.184708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.131:48843)
0620 11:33:55.184906 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:55.184919 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:55.185019 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:55.185202 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:55.185203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:55.185388 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:55.185436 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:55.185439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:55.186170 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:55.186171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:55.186702 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:55.186704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:55.187202 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:55.187203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:55.187260 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:55.187763 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:55.187778 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:55.189532 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":105,"threads_started":1}
W20260620 11:33:55.189673 12184 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:55.184819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:48843 (local address 127.2.243.190:33141)
0620 11:33:55.184910 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:55.184912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:55.185000 (+ 88us) server_negotiation.cc:407] Connection header received
0620 11:33:55.185093 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:55.185094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:55.185112 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:55.185196 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:55.185529 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:55.186083 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:55.186816 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:55.187112 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:55.187201 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:55.189613 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.131:48843: BlockingRecv error: recv got EOF from 127.2.243.131:48843 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 11:33:55 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:55 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:55.940953 12187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:55.936360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.129:33569)
0620 11:33:55.936503 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:55.936521 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:55.936581 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:55.936744 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:55.936745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:55.936920 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:55.936968 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:55.936970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:55.937612 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:55.937613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:55.938125 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:55.938126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:55.938444 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:55.938445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:55.938485 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:55.939092 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:55.939097 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:55.940893 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
W20260620 11:33:55.941099 12188 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:55.936412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:33569 (local address 127.2.243.190:33141)
0620 11:33:55.936536 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:55.936538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:55.936576 (+ 38us) server_negotiation.cc:407] Connection header received
0620 11:33:55.936651 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:55.936651 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:55.936661 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:55.936727 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:55.937028 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:55.937523 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:55.938193 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:55.938378 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:55.938432 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:55.940999 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.129:33569: BlockingRecv error: recv got EOF from 127.2.243.129:33569 (error 108)
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
Jun 20 11:33:56 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:56 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:56.067508 12189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:56.062337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.130:46957)
0620 11:33:56.062542 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:56.062554 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:56.062634 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:56.062783 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:56.062784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:56.063001 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:56.063046 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:56.063047 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:56.063724 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:56.063726 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:56.064273 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:56.064274 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:56.064741 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:56.064741 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:56.064797 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:56.065279 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:56.065283 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:56.067406 (+ 2123us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":105,"threads_started":1}
W20260620 11:33:56.067597 12188 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:56.062429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:46957 (local address 127.2.243.190:33141)
0620 11:33:56.062521 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:56.062523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:56.062627 (+ 104us) server_negotiation.cc:407] Connection header received
0620 11:33:56.062708 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:56.062709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:56.062722 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:56.062768 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:56.063133 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:56.063637 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:56.064378 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:56.064652 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:56.064739 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:56.067511 (+ 2772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.130:46957: BlockingRecv error: recv got EOF from 127.2.243.130:46957 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 11:33:56 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:56 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:56.195168 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:56.190254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.131:37541)
0620 11:33:56.190461 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:56.190474 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:56.190564 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:56.190723 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:56.190724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:56.190997 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:56.191058 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:56.191061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:56.191808 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:56.191809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:56.192342 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:56.192344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:56.192716 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:56.192717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:56.192772 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:56.193264 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:56.193268 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:56.195110 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":106,"threads_started":1}
W20260620 11:33:56.195302 12188 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:56.190381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:37541 (local address 127.2.243.190:33141)
0620 11:33:56.190460 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:56.190462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:56.190561 (+ 99us) server_negotiation.cc:407] Connection header received
0620 11:33:56.190623 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:56.190624 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:56.190637 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:56.190715 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:56.191128 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:56.191726 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:56.192435 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:56.192633 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:56.192716 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:56.195219 (+ 2503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.131:37541: BlockingRecv error: recv got EOF from 127.2.243.131:37541 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 11:33:56 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:56 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:56.946276 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:56.941468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.129:53957)
0620 11:33:56.941600 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:56.941616 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:56.941669 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:56.941825 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:56.941826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:56.941985 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:56.942037 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:56.942039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:56.942694 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:56.942695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:56.943269 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:56.943271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:56.943628 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:56.943629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:56.943681 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:56.944302 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:56.944307 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:56.946216 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":79,"threads_started":1}
W20260620 11:33:56.946416 12192 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:56.941553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:53957 (local address 127.2.243.190:33141)
0620 11:33:56.941699 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:56.941701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:56.941709 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:33:56.941731 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:56.941732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:56.941742 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:56.941806 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:56.942128 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:56.942616 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:56.943346 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:56.943534 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:56.943590 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:56.946307 (+ 2717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.129:53957: BlockingRecv error: recv got EOF from 127.2.243.129:53957 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
Jun 20 11:33:57 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:57 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:57.076231 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:57.068090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.130:39295)
0620 11:33:57.068239 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:57.068253 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:57.068312 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:57.068617 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:57.068618 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:57.068859 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:57.068919 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:57.068921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:57.069561 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:57.069561 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:57.070203 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:57.070205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:57.070587 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:57.070588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:57.070642 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:57.074220 (+ 3578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:57.074225 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:57.076170 (+ 1945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":67,"threads_started":1}
W20260620 11:33:57.076377 12192 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:57.068229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:39295 (local address 127.2.243.190:33141)
0620 11:33:57.068281 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:57.068285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:57.068300 (+ 15us) server_negotiation.cc:407] Connection header received
0620 11:33:57.068495 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:57.068496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:57.068508 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:57.068602 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:57.068994 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:57.069479 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:57.070297 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:57.070496 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:57.070587 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:57.076287 (+ 5700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.130:39295: BlockingRecv error: recv got EOF from 127.2.243.130:39295 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 11:33:57 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:57 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:57.200584 12194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:57.195682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.131:49549)
0620 11:33:57.195876 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:57.195890 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:57.195958 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:57.196189 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:57.196190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:57.196354 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:57.196401 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:57.196403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:57.197044 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:57.197044 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:57.197555 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:57.197557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:57.198003 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:57.198004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:57.198065 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:57.198541 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:57.198546 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:57.200488 (+ 1942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":61,"threads_started":1}
W20260620 11:33:57.200624 12192 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:57.195866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:49549 (local address 127.2.243.190:33141)
0620 11:33:57.195926 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:57.195929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:57.195951 (+ 22us) server_negotiation.cc:407] Connection header received
0620 11:33:57.196019 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:57.196020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:57.196032 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:57.196140 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:57.196475 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:57.196973 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:57.197703 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:57.197893 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:57.197990 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:57.200564 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.131:49549: BlockingRecv error: recv got EOF from 127.2.243.131:49549 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 20 11:33:57 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:57 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:57.951372 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:57.946814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.129:54049)
0620 11:33:57.946978 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:57.946997 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:57.947049 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:57.947230 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:57.947231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:57.947394 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:57.947441 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:57.947442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:57.948103 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:57.948104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:57.948697 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:57.948699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:57.949036 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:57.949037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:57.949075 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:57.949550 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:57.949554 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:57.951314 (+ 1760us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":108,"threads_started":1}
W20260620 11:33:57.951565 12196 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:57.946929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:54049 (local address 127.2.243.190:33141)
0620 11:33:57.947096 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:57.947098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:57.947107 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:33:57.947128 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:57.947129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:57.947139 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:57.947205 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:57.947520 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:57.948024 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:57.948781 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:57.948968 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:57.949023 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:57.951467 (+ 2444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.129:54049: BlockingRecv error: recv got EOF from 127.2.243.129:54049 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":113,"threads_started":1}
Jun 20 11:33:58 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:58 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:58.081663 12197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:58.076735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.130:57583)
0620 11:33:58.076912 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:58.076924 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:58.076989 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:58.077128 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:58.077129 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:58.077335 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:58.077384 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:58.077386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:58.078051 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:58.078052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:58.078611 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:58.078613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:58.079042 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:58.079043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:58.079088 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:58.079684 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:58.079689 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:58.081564 (+ 1875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":129,"threads_started":1}
W20260620 11:33:58.081715 12196 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:58.076826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:57583 (local address 127.2.243.190:33141)
0620 11:33:58.076915 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:58.076917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:58.076993 (+ 76us) server_negotiation.cc:407] Connection header received
0620 11:33:58.077050 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:58.077051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:58.077062 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:58.077104 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:58.077451 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:58.077950 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:58.078696 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:58.078955 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:58.079027 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:58.081651 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.130:57583: BlockingRecv error: recv got EOF from 127.2.243.130:57583 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 11:33:58 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:58 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:58.205942 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:58.201139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.131:58951)
0620 11:33:58.201269 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:58.201282 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:58.201352 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:58.201501 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:58.201502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:58.201661 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:58.201708 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:58.201710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:58.202386 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:58.202387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:58.202939 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:58.202941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:58.203395 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:58.203396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:58.203432 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:58.203873 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:58.203877 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:58.205841 (+ 1964us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
W20260620 11:33:58.205969 12196 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:58.201229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:58951 (local address 127.2.243.190:33141)
0620 11:33:58.201328 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:58.201330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:58.201341 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:33:58.201423 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:58.201424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:58.201435 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:58.201486 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:58.201767 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:58.202316 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:58.203057 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:58.203323 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:58.203393 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:58.205912 (+ 2519us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.131:58951: BlockingRecv error: recv got EOF from 127.2.243.131:58951 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 20 11:33:58 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:58 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:58.956794 12199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:58.951894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.129:41705)
0620 11:33:58.952058 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:58.952069 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:58.952139 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:58.952281 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:58.952281 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:58.952445 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:58.952490 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:58.952492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:58.953133 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:58.953133 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:58.953950 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:58.953969 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:58.954315 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:58.954316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:58.954356 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:58.954792 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:58.954797 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:58.956695 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
W20260620 11:33:58.956872 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:58.951979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:41705 (local address 127.2.243.190:33141)
0620 11:33:58.952080 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:58.952082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:58.952137 (+ 55us) server_negotiation.cc:407] Connection header received
0620 11:33:58.952196 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:58.952197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:58.952206 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:58.952257 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:58.952564 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:58.953066 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:58.954038 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:58.954232 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:58.954285 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:58.956775 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.129:41705: BlockingRecv error: recv got EOF from 127.2.243.129:41705 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":50,"threads_started":1}
Jun 20 11:33:59 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:59 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:59.086949 12201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:59.082218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.130:59231)
0620 11:33:59.082407 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:59.082429 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:59.082494 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:59.082649 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:59.082649 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:59.082825 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:59.082950 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:59.082952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:59.083615 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:59.083616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:59.084136 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:59.084137 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:59.084465 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:59.084466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:59.084508 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:59.084992 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:59.084996 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:59.086828 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":76,"threads_started":1}
W20260620 11:33:59.086992 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:59.082436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:59231 (local address 127.2.243.190:33141)
0620 11:33:59.082506 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:59.082508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:59.082544 (+ 36us) server_negotiation.cc:407] Connection header received
0620 11:33:59.082570 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:59.082570 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:59.082580 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:59.082623 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:59.083024 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:59.083526 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:59.084213 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:59.084397 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:59.084448 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:59.086944 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.130:59231: BlockingRecv error: recv got EOF from 127.2.243.130:59231 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 11:33:59 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:59 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:59.211267 12202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:59.206442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.131:54671)
0620 11:33:59.206568 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:59.206581 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:59.206661 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:59.206807 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:59.206808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:59.206984 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:59.207031 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:59.207032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:59.207667 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:59.207668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:59.208214 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:59.208216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:59.208660 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:59.208660 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:59.208712 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:59.209193 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:59.209198 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:59.211207 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
W20260620 11:33:59.211381 12200 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:59.206540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:54671 (local address 127.2.243.190:33141)
0620 11:33:59.206618 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:59.206620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:59.206643 (+ 23us) server_negotiation.cc:407] Connection header received
0620 11:33:59.206718 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:59.206719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:59.206728 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:59.206779 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:59.207094 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:59.207585 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:59.208320 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:59.208523 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:59.208613 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:59.211303 (+ 2690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.131:54671: BlockingRecv error: recv got EOF from 127.2.243.131:54671 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 11:33:59 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:33:59 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:33:59.962433 12203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:33:59.957487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.129:51487)
0620 11:33:59.957737 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 11:33:59.957748 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:33:59.957823 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:33:59.958105 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:33:59.958106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:33:59.958316 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:33:59.958386 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:59.958388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:59.959092 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:59.959093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:59.959624 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:33:59.959626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:33:59.959949 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:33:59.959950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:33:59.959993 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:59.960487 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:33:59.960492 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:33:59.962374 (+ 1882us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":72,"threads_started":1}
W20260620 11:33:59.962597 12204 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:33:59.957617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:51487 (local address 127.2.243.190:33141)
0620 11:33:59.957727 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 11:33:59.957729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:33:59.957824 (+ 95us) server_negotiation.cc:407] Connection header received
0620 11:33:59.957899 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:33:59.957900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:33:59.957926 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:33:59.958017 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:33:59.958462 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:59.958984 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:59.959697 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:33:59.959881 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:33:59.959930 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:33:59.962502 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.129:51487: BlockingRecv error: recv got EOF from 127.2.243.129:51487 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":45,"threads_started":1}
Jun 20 11:34:00 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:34:00 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:34:00.092846 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:00.087529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.130:36325)
0620 11:34:00.087774 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:00.087786 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:00.087892 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:00.088082 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:00.088083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:00.088262 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:00.088308 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:00.088310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:00.089337 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:00.089338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:00.089872 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:00.089874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:00.090268 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:00.090268 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:00.090327 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:00.090842 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:00.090847 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:00.092744 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":117,"threads_started":1}
W20260620 11:34:00.092890 12204 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:00.087642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:36325 (local address 127.2.243.190:33141)
0620 11:34:00.087762 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:00.087765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:00.087874 (+ 109us) server_negotiation.cc:407] Connection header received
0620 11:34:00.087954 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:00.087956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:00.087968 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:00.088074 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:00.088392 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:00.089257 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:00.089962 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:00.090178 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:00.090265 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:00.092825 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.130:36325: BlockingRecv error: recv got EOF from 127.2.243.130:36325 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 20 11:34:00 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:34:00 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:34:00.216288 12206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:00.211794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.131:53883)
0620 11:34:00.211956 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:00.211967 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:00.212025 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:00.212157 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:00.212158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:00.212314 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:00.212360 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:00.212361 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:00.212962 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:00.212963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:00.213489 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:00.213491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:00.213829 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:00.213830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:00.213881 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:00.214334 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:00.214340 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:00.216189 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":89,"threads_started":1}
W20260620 11:34:00.216329 12204 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:00.211844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:53883 (local address 127.2.243.190:33141)
0620 11:34:00.211918 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:00.211920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:00.212028 (+ 108us) server_negotiation.cc:407] Connection header received
0620 11:34:00.212082 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:00.212083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:00.212094 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:00.212135 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:00.212420 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:00.212900 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:00.213563 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:00.213741 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:00.213793 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:00.216268 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.131:53883: BlockingRecv error: recv got EOF from 127.2.243.131:53883 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 11:34:00 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:34:00 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:34:00.968003 12207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:00.963100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.129:52625)
0620 11:34:00.963233 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:00.963243 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:00.963294 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:00.963488 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:00.963489 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:00.963719 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:00.963796 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:00.963799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:00.964594 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:00.964595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:00.965120 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:00.965122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:00.965479 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:00.965480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:00.965521 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:00.966154 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:00.966159 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:00.967944 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
W20260620 11:34:00.968169 12208 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:00.963163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:52625 (local address 127.2.243.190:33141)
0620 11:34:00.963311 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:00.963313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:00.963342 (+ 29us) server_negotiation.cc:407] Connection header received
0620 11:34:00.963371 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:00.963371 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:00.963385 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:00.963472 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:00.963877 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:00.964523 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:00.965194 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:00.965381 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:00.965464 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:00.968063 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.129:52625: BlockingRecv error: recv got EOF from 127.2.243.129:52625 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
Jun 20 11:34:01 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:34:01 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:34:01.098407 12209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:01.093379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.130:45779)
0620 11:34:01.093609 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:01.093620 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:01.093718 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:01.093860 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:01.093860 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:01.094023 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:01.094103 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:01.094105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:01.095019 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:01.095020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:01.095575 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:01.095577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:01.095891 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:01.095892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:01.095932 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:01.096383 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:01.096387 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:01.098321 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":104,"threads_started":1}
W20260620 11:34:01.098503 12208 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:01.093518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:45779 (local address 127.2.243.190:33141)
0620 11:34:01.093578 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:01.093580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:01.093701 (+ 121us) server_negotiation.cc:407] Connection header received
0620 11:34:01.093775 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:01.093776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:01.093786 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:01.093836 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:01.094203 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:01.094945 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:01.095650 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:01.095827 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:01.095876 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:01.098412 (+ 2536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.130:45779: BlockingRecv error: recv got EOF from 127.2.243.130:45779 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Jun 20 11:34:01 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:34:01 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:34:01.221457 12210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:01.216753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.131:50895)
0620 11:34:01.216873 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:01.216884 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:01.216950 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:01.217084 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:01.217085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:01.217278 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:01.217326 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:01.217327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:01.218057 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:01.218058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:01.218591 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:01.218593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:01.218966 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:01.218967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:01.219027 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:01.219554 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:01.219566 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:01.221398 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
W20260620 11:34:01.221625 12208 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:01.216844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:50895 (local address 127.2.243.190:33141)
0620 11:34:01.216908 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:01.216910 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:01.216940 (+ 30us) server_negotiation.cc:407] Connection header received
0620 11:34:01.217004 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:01.217004 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:01.217015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:01.217073 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:01.217426 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:01.217973 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:01.218679 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:01.218890 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:01.218951 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:01.221557 (+ 2606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.131:50895: BlockingRecv error: recv got EOF from 127.2.243.131:50895 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 11:34:01 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:34:01 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:34:01.973623 12211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:01.968532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.129:33607)
0620 11:34:01.968717 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:01.968728 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:01.968786 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:01.968939 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:01.968939 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:01.969100 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:01.969144 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:01.969145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:01.969848 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:01.969849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:01.970537 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:01.970540 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:01.971068 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:01.971069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:01.971150 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:01.971644 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:01.971648 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:01.973510 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":107,"threads_started":1}
W20260620 11:34:01.973660 12212 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:01.968596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:33607 (local address 127.2.243.190:33141)
0620 11:34:01.968768 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:01.968769 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:01.968781 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:34:01.968844 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:01.968845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:01.968856 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:01.968924 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:01.969210 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:01.969747 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:01.970620 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:01.970980 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:01.971059 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:01.973589 (+ 2530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.129:33607: BlockingRecv error: recv got EOF from 127.2.243.129:33607 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":85,"threads_started":1}
Jun 20 11:34:02 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:34:02 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:34:02.107024 12213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:02.098955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.130:36783)
0620 11:34:02.099082 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:02.099124 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:02.099221 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:02.099368 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:02.099369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:02.099544 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:02.099590 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:02.099591 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:02.100265 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:02.100267 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:02.100801 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:02.100803 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:02.101184 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:02.101185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:02.101248 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:02.104959 (+ 3711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:02.104965 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:02.106928 (+ 1963us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
W20260620 11:34:02.107061 12212 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:02.099067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:36783 (local address 127.2.243.190:33141)
0620 11:34:02.099153 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:02.099155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:02.099213 (+ 58us) server_negotiation.cc:407] Connection header received
0620 11:34:02.099293 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:02.099294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:02.099306 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:02.099354 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:02.099677 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:02.100181 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:02.100890 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:02.101089 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:02.101176 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:02.107001 (+ 5825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.130:36783: BlockingRecv error: recv got EOF from 127.2.243.130:36783 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 11:34:02 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:34:02 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:34:02.227485 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:02.222084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.131:38607)
0620 11:34:02.222253 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:02.222268 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:02.222339 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:02.222498 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:02.222499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:02.222664 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:02.222712 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:02.222714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:02.223416 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:02.223418 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:02.224345 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:02.224349 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:02.224800 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:02.224801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:02.224850 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:02.225420 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:02.225424 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:02.227423 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":71,"threads_started":1}
W20260620 11:34:02.227587 12212 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:02.222243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:38607 (local address 127.2.243.190:33141)
0620 11:34:02.222311 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:02.222313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:02.222326 (+ 13us) server_negotiation.cc:407] Connection header received
0620 11:34:02.222400 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:02.222401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:02.222412 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:02.222465 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:02.222812 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:02.223315 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:02.224459 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:02.224657 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:02.224758 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:02.227518 (+ 2760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.131:38607: BlockingRecv error: recv got EOF from 127.2.243.131:38607 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 20 11:34:02 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:34:02 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:34:02.979148 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:02.974172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.129:58799)
0620 11:34:02.974314 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:02.974332 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:02.974386 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:02.974574 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:02.974575 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:02.974739 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:02.974791 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:02.974792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:02.975551 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:02.975553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:02.976099 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:02.976101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:02.976446 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:02.976446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:02.976487 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:02.976968 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:02.976972 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:02.979053 (+ 2081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
W20260620 11:34:02.979310 12216 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:02.974251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:58799 (local address 127.2.243.190:33141)
0620 11:34:02.974399 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:02.974401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:02.974435 (+ 34us) server_negotiation.cc:407] Connection header received
0620 11:34:02.974459 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:02.974460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:02.974471 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:02.974525 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:02.974910 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:02.975460 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:02.976180 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:02.976380 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:02.976433 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:02.979185 (+ 2752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.129:58799: BlockingRecv error: recv got EOF from 127.2.243.129:58799 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
Jun 20 11:34:03 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:34:03 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:34:03.112802 12217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:03.107592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.130:35597)
0620 11:34:03.107744 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:03.107756 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:03.107829 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:03.108022 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:03.108023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:03.108190 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:03.108235 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:03.108236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:03.108967 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:03.108968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:03.109540 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:03.109542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:03.110165 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:03.110166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:03.110241 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:03.110747 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:03.110753 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:03.112730 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":104,"threads_started":1}
W20260620 11:34:03.112931 12216 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:03.107649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:35597 (local address 127.2.243.190:33141)
0620 11:34:03.107738 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:03.107741 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:03.107814 (+ 73us) server_negotiation.cc:407] Connection header received
0620 11:34:03.107906 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:03.107907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:03.107919 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:03.107971 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:03.108341 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:03.108857 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:03.109661 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:03.109976 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:03.110099 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:03.112853 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.130:35597: BlockingRecv error: recv got EOF from 127.2.243.130:35597 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 11:34:03 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:34:03 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:34:03.232755 12218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:03.228016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.131:33015)
0620 11:34:03.228161 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:03.228172 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:03.228235 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:03.228375 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:03.228376 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:03.228536 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:03.228585 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:03.228586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:03.229212 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:03.229213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:03.229732 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:03.229734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:03.230067 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:03.230068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:03.230113 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:03.230641 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:03.230646 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:03.232695 (+ 2049us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":82,"threads_started":1}
W20260620 11:34:03.232852 12216 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:03.228103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:33015 (local address 127.2.243.190:33141)
0620 11:34:03.228173 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:03.228175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:03.228221 (+ 46us) server_negotiation.cc:407] Connection header received
0620 11:34:03.228297 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:03.228298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:03.228308 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:03.228353 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:03.228649 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:03.229144 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:03.229806 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:03.229996 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:03.230048 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:03.232784 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.131:33015: BlockingRecv error: recv got EOF from 127.2.243.131:33015 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 11:34:03 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:34:03 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:34:03.984292 12219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:03.979657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.129:39597)
0620 11:34:03.979818 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:03.979836 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:03.979914 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:03.980063 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:03.980064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:03.980224 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:03.980270 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:03.980272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:03.980900 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:03.980900 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:03.981429 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:03.981431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:03.981786 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:03.981787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:03.981825 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:03.982278 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:03.982282 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:03.984192 (+ 1910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
W20260620 11:34:03.984495 12220 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:03.979725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:39597 (local address 127.2.243.190:33141)
0620 11:34:03.979937 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:03.979939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:03.979948 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:34:03.979972 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:03.979973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:03.979983 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:03.980044 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:03.980335 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:03.980827 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:03.981525 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:03.981718 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:03.981774 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:03.984313 (+ 2539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.129:39597: BlockingRecv error: recv got EOF from 127.2.243.129:39597 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":124,"threads_started":1}
Jun 20 11:34:04 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:34:04 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:34:04.118242 12221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:04.113330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.130:57935)
0620 11:34:04.113563 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:04.113574 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:04.113644 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:04.113787 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:04.113788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:04.113952 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:04.114004 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:04.114006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:04.114622 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:04.114623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:04.115191 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:04.115193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:04.115546 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:04.115547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:04.115586 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:04.116147 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:04.116152 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:04.118182 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":179,"threads_started":1}
W20260620 11:34:04.118429 12220 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:04.113451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:57935 (local address 127.2.243.190:33141)
0620 11:34:04.113535 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:04.113538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:04.113646 (+ 108us) server_negotiation.cc:407] Connection header received
0620 11:34:04.113709 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:04.113710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:04.113720 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:04.113766 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:04.114069 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:04.114551 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:04.115281 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:04.115474 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:04.115535 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:04.118314 (+ 2779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.130:57935: BlockingRecv error: recv got EOF from 127.2.243.130:57935 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 11:34:04 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:34:04 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:34:04.238384 12222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:04.233316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.131:53153)
0620 11:34:04.233495 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:04.233507 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:04.233566 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:04.233704 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:04.233705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:04.233873 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:04.233918 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:04.233920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:04.234691 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:04.234692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:04.235295 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:04.235297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:04.235645 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:04.235646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:04.235684 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:04.236132 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:04.236136 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:04.238293 (+ 2157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":98,"threads_started":1}
W20260620 11:34:04.238539 12220 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:04.233388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:53153 (local address 127.2.243.190:33141)
0620 11:34:04.233489 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:04.233491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:04.233567 (+ 76us) server_negotiation.cc:407] Connection header received
0620 11:34:04.233613 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:04.233614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:04.233626 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:04.233684 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:04.234018 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:04.234617 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:04.235378 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:04.235579 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:04.235635 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:04.238458 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.131:53153: BlockingRecv error: recv got EOF from 127.2.243.131:53153 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 11:34:04 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:34:04 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:34:04.989966 12223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:04.984848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.129:58187)
0620 11:34:04.985010 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:04.985023 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:04.985085 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:04.985297 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:04.985297 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:04.985536 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:04.985616 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:04.985618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:04.986551 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:04.986552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:04.987255 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:04.987258 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:04.987642 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:04.987643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:04.987691 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:04.988179 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:04.988183 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:04.989906 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
W20260620 11:34:04.990136 12224 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:04.984980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:58187 (local address 127.2.243.190:33141)
0620 11:34:04.985130 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:04.985132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:04.985142 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:34:04.985168 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:04.985168 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:04.985178 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:04.985283 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:04.985719 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:04.986451 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:04.987362 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:04.987560 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:04.987633 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:04.990012 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.129:58187: BlockingRecv error: recv got EOF from 127.2.243.129:58187 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
Jun 20 11:34:05 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:34:05 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:34:05.123607 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:05.118786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.130:35493)
0620 11:34:05.119068 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:05.119081 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:05.119146 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:05.119314 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:05.119315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:05.119482 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:05.119529 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:05.119530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:05.120205 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:05.120206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:05.120747 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:05.120749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:05.121122 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:05.121123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:05.121175 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:05.121681 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:05.121686 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:05.123544 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":209,"threads_started":1}
W20260620 11:34:05.123749 12224 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:05.118976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:35493 (local address 127.2.243.190:33141)
0620 11:34:05.119023 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:05.119025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:05.119151 (+ 126us) server_negotiation.cc:407] Connection header received
0620 11:34:05.119221 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:05.119222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:05.119234 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:05.119292 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:05.119591 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:05.120083 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:05.120852 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:05.121040 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:05.121122 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:05.123644 (+ 2522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.130:35493: BlockingRecv error: recv got EOF from 127.2.243.130:35493 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 11:34:05 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:34:05 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:34:05.243623 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:05.238913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.131:33281)
0620 11:34:05.239111 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:05.239122 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:05.239187 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:05.239332 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:05.239333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:05.239510 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:05.239561 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:05.239563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:05.240241 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:05.240242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:05.240782 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:05.240784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:05.241119 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:05.241120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:05.241164 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:05.241622 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:05.241627 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:05.243561 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":79,"threads_started":1}
W20260620 11:34:05.243728 12224 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:05.238983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:33281 (local address 127.2.243.190:33141)
0620 11:34:05.239045 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:05.239047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:05.239185 (+ 138us) server_negotiation.cc:407] Connection header received
0620 11:34:05.239235 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:05.239236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:05.239247 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:05.239301 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:05.239641 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:05.240141 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:05.240865 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:05.241045 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:05.241101 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:05.243656 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.131:33281: BlockingRecv error: recv got EOF from 127.2.243.131:33281 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Jun 20 11:34:05 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:34:05 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:34:05.995679 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:05.990562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.129:52705)
0620 11:34:05.990713 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:05.990725 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:05.990790 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:05.991014 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:05.991015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:05.991221 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:05.991267 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:05.991269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:05.992145 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:05.992146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:05.992754 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:05.992756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:05.993217 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:05.993218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:05.993260 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:05.993810 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:05.993815 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:05.995619 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
W20260620 11:34:05.995847 12228 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:05.990638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:52705 (local address 127.2.243.190:33141)
0620 11:34:05.990801 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:05.990803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:05.990845 (+ 42us) server_negotiation.cc:407] Connection header received
0620 11:34:05.990913 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:05.990914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:05.990933 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:05.990999 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:05.991350 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:05.992070 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:05.992845 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:05.993053 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:05.993182 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:05.995763 (+ 2581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.129:52705: BlockingRecv error: recv got EOF from 127.2.243.129:52705 (error 108)
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
Jun 20 11:34:06 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:34:06 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:34:06.128636 12229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:06.124070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.130:54939)
0620 11:34:06.124289 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:06.124307 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:06.124369 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:06.124497 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:06.124498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:06.124668 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:06.124727 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:06.124729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:06.125343 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:06.125344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:06.125901 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:06.125902 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:06.126225 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:06.126226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:06.126270 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:06.126720 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:06.126725 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:06.128565 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":156,"threads_started":1}
W20260620 11:34:06.128806 12228 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:06.124179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:54939 (local address 127.2.243.190:33141)
0620 11:34:06.124244 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:06.124246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:06.124364 (+ 118us) server_negotiation.cc:407] Connection header received
0620 11:34:06.124416 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:06.124417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:06.124427 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:06.124479 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:06.124797 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:06.125278 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:06.125974 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:06.126156 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:06.126207 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:06.128687 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.130:54939: BlockingRecv error: recv got EOF from 127.2.243.130:54939 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 11:34:06 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:34:06 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.131@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:34:06.249351 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:06.244392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.131:43405)
0620 11:34:06.244545 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:06.244557 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:06.244644 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:06.244867 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:06.244868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:06.245073 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:06.245120 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:06.245122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:06.245813 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:06.245814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:06.246427 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:06.246429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:06.246814 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:06.246815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:06.246940 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:06.247409 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:06.247413 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:06.249292 (+ 1879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
W20260620 11:34:06.249500 12228 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:06.244492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:43405 (local address 127.2.243.190:33141)
0620 11:34:06.244543 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:06.244546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:06.244640 (+ 94us) server_negotiation.cc:407] Connection header received
0620 11:34:06.244710 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:06.244710 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:06.244726 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:06.244779 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:06.245199 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:06.245717 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:06.246527 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:06.246711 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:06.246780 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:06.249413 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.131:43405: BlockingRecv error: recv got EOF from 127.2.243.131:43405 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 20 11:34:06 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:34:06 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.129@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:34:07.001070 12231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:06.996252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.129:54009)
0620 11:34:06.996418 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:06.996429 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:06.996479 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:06.996662 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:06.996664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:06.996828 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:06.996880 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:06.996882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:06.997561 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:06.997562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:06.998111 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:06.998113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:06.998470 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:06.998471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:06.998511 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:06.999058 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:06.999063 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:07.001011 (+ 1948us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":104,"threads_started":1}
W20260620 11:34:07.001205 12232 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:06.996372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:54009 (local address 127.2.243.190:33141)
0620 11:34:06.996505 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:06.996507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:06.996515 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:34:06.996538 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:06.996538 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:06.996548 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:06.996614 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:06.996957 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:06.997476 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:06.998197 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:06.998401 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:06.998458 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:07.001114 (+ 2656us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.129:54009: BlockingRecv error: recv got EOF from 127.2.243.129:54009 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
Jun 20 11:34:07 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:34:07 dist-test-slave-ctbw krb5kdc[11172](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.2.243.130@KRBTEST.COM for oryx/127.2.243.190@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:34:07.139010 12233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:07.129184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33141 (local address 127.2.243.130:41933)
0620 11:34:07.129339 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:07.129354 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:07.129426 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:07.129560 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:07.129561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:07.129735 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:07.129805 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:07.129806 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:07.130470 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:07.130471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:07.131157 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:07.131159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:07.131549 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:07.131550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:07.131598 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:07.136760 (+ 5162us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:07.136783 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:07.138945 (+ 2162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.2.243.190:33141: Server oryx/127.2.243.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
W20260620 11:34:07.139078 12232 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:07.129330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:41933 (local address 127.2.243.190:33141)
0620 11:34:07.129403 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:07.129405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:07.129416 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:34:07.129489 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:07.129490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:07.129501 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:07.129553 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:07.129868 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:07.130385 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:07.131249 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:07.131454 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:07.131518 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:07.139029 (+ 7511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.2.243.130:41933: BlockingRecv error: recv got EOF from 127.2.243.130:41933 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 11:34:07.166236 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 11756
I20260620 11:34:07.171120 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 11907
I20260620 11:34:07.175484 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 12040
I20260620 11:34:07.180380 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 11685
2026-06-20T11:34:07Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17425 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T11:34:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:34:07Z Disabled control of system clock
I20260620 11:34:07.208520 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:43919
--webserver_interface=127.2.243.190
--webserver_port=0
--builtin_ntp_servers=127.2.243.148:32951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:43919
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 11:34:07.290350 12237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:34:07.290565 12237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:34:07.290587 12237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:34:07.291736 12237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:34:07.291774 12237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:34:07.291787 12237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:34:07.291800 12237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:34:07.291812 12237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:34:07.293220 12237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:32951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:43919
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:43919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:34:07.293426 12237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:34:07.293677 12237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:34:07.298828 12243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:34:07.298884 12237 server_base.cc:1034] running on GCE node
W20260620 11:34:07.298840 12245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:07.298840 12242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:34:07.299253 12237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:34:07.299554 12237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:34:07.300709 12237 hybrid_clock.cc:648] HybridClock initialized: now 1781955247300700 us; error 32 us; skew 500 ppm
I20260620 11:34:07.301946 12237 webserver.cc:460] Webserver started at http://127.2.243.190:33849/ using document root <none> and password file <none>
I20260620 11:34:07.302217 12237 fs_manager.cc:359] Metadata directory not provided
I20260620 11:34:07.302289 12237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:34:07.302507 12237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:34:07.303653 12237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "be4d9cfb0bdc49eba2f893546ccd3498"
format_stamp: "Formatted at 2026-06-20 11:34:07 on dist-test-slave-ctbw"
server_key: "bb0c241ecea78dea58df21aa3745a0c5"
server_key_iv: "4ee44ee5b1e58373974daa25837e1987"
server_key_version: "encryptionkey@0"
I20260620 11:34:07.304033 12237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "be4d9cfb0bdc49eba2f893546ccd3498"
format_stamp: "Formatted at 2026-06-20 11:34:07 on dist-test-slave-ctbw"
server_key: "bb0c241ecea78dea58df21aa3745a0c5"
server_key_iv: "4ee44ee5b1e58373974daa25837e1987"
server_key_version: "encryptionkey@0"
I20260620 11:34:07.305424 12237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 11:34:07.306175 12251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:07.306439 12237 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:34:07.306536 12237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "be4d9cfb0bdc49eba2f893546ccd3498"
format_stamp: "Formatted at 2026-06-20 11:34:07 on dist-test-slave-ctbw"
server_key: "bb0c241ecea78dea58df21aa3745a0c5"
server_key_iv: "4ee44ee5b1e58373974daa25837e1987"
server_key_version: "encryptionkey@0"
I20260620 11:34:07.306598 12237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:34:07.320541 12237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:34:07.320860 12237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:34:07.320977 12237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:34:07.325165 12237 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:43919
I20260620 11:34:07.325220 12303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:43919 every 8 connection(s)
I20260620 11:34:07.325605 12237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:34:07.326349 12304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:34:07.328986 12304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498: Bootstrap starting.
I20260620 11:34:07.329603 12304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498: Neither blocks nor log segments found. Creating new log.
I20260620 11:34:07.329919 12304 log.cc:824] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498: Log is configured to *not* fsync() on all Append() calls
I20260620 11:34:07.330600 12304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498: No bootstrap required, opened a new log
I20260620 11:34:07.332249 12304 raft_consensus.cc:348] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4d9cfb0bdc49eba2f893546ccd3498" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43919 } }
I20260620 11:34:07.332389 12304 raft_consensus.cc:374] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:34:07.332414 12304 raft_consensus.cc:729] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be4d9cfb0bdc49eba2f893546ccd3498, State: Initialized, Role: FOLLOWER
I20260620 11:34:07.332597 12304 consensus_queue.cc:260] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4d9cfb0bdc49eba2f893546ccd3498" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43919 } }
I20260620 11:34:07.332660 12304 raft_consensus.cc:388] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:34:07.332701 12304 raft_consensus.cc:482] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:34:07.332751 12304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:34:07.332960 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 12237
I20260620 11:34:07.333042 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:34:07.333238 3022 external_mini_cluster.cc:1428] Setting key 91260e34e48da7c072f50b801d6f8aef
I20260620 11:34:07.333389 12304 raft_consensus.cc:504] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4d9cfb0bdc49eba2f893546ccd3498" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43919 } }
I20260620 11:34:07.333511 12304 leader_election.cc:302] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be4d9cfb0bdc49eba2f893546ccd3498; no voters:
I20260620 11:34:07.333699 12304 leader_election.cc:288] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:34:07.333750 12307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:34:07.334246 12304 sys_catalog.cc:564] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:34:07.334244 12307 raft_consensus.cc:686] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498 [term 1 LEADER]: Becoming Leader. State: Replica: be4d9cfb0bdc49eba2f893546ccd3498, State: Running, Role: LEADER
I20260620 11:34:07.334509 12307 consensus_queue.cc:237] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4d9cfb0bdc49eba2f893546ccd3498" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43919 } }
I20260620 11:34:07.334964 12311 sys_catalog.cc:455] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be4d9cfb0bdc49eba2f893546ccd3498. Latest consensus state: current_term: 1 leader_uuid: "be4d9cfb0bdc49eba2f893546ccd3498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4d9cfb0bdc49eba2f893546ccd3498" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43919 } } }
I20260620 11:34:07.335039 12311 sys_catalog.cc:458] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498 [sys.catalog]: This master's current role is: LEADER
I20260620 11:34:07.335377 12310 sys_catalog.cc:455] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be4d9cfb0bdc49eba2f893546ccd3498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4d9cfb0bdc49eba2f893546ccd3498" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43919 } } }
I20260620 11:34:07.335443 12310 sys_catalog.cc:458] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498 [sys.catalog]: This master's current role is: LEADER
I20260620 11:34:07.335692 12318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:34:07.336166 12318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:34:07.337759 12318 catalog_manager.cc:1347] Generated new cluster ID: f0e5868dd30b493ca21f7806c2ef9cf7
I20260620 11:34:07.337814 12318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:34:07.339718 12309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:07.333826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33040 (local address 127.2.243.190:43919)
0620 11:34:07.334057 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:07.334068 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:07.334129 (+ 61us) server_negotiation.cc:407] Connection header received
0620 11:34:07.334247 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:07.334254 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:07.334329 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:07.334499 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:07.334740 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:07.336612 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:07.337707 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:07.338169 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:07.338300 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:07.339079 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:34:07.339083 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:34:07.339083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:34:07.339266 (+ 183us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:34:07.339306 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:34:07.339317 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:34:07.339406 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:07.339478 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 11:34:07.339557 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":114,"threads_started":1}
I20260620 11:34:07.347700 12318 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:34:07.348249 12318 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:34:07.350497 12318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be4d9cfb0bdc49eba2f893546ccd3498: Generated new TSK 0
I20260620 11:34:07.350682 12318 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:34:07.352387 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:43919
--builtin_ntp_servers=127.2.243.148:32951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 11:34:07.452769 12328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:34:07.452973 12328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:34:07.452994 12328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:34:07.454231 12328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:34:07.454272 12328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:34:07.454298 12328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:34:07.455689 12328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:32951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:43919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:34:07.455929 12328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:34:07.456247 12328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:34:07.461256 12333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:07.461308 12336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:07.461256 12334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:34:07.461452 12328 server_base.cc:1034] running on GCE node
I20260620 11:34:07.461808 12328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:34:07.462061 12328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:34:07.463218 12328 hybrid_clock.cc:648] HybridClock initialized: now 1781955247463205 us; error 33 us; skew 500 ppm
I20260620 11:34:07.464496 12328 webserver.cc:460] Webserver started at http://127.2.243.129:33081/ using document root <none> and password file <none>
I20260620 11:34:07.464742 12328 fs_manager.cc:359] Metadata directory not provided
I20260620 11:34:07.464813 12328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:34:07.464972 12328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:34:07.466010 12328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "7c8092641bd54eea84d8aa80751bf390"
format_stamp: "Formatted at 2026-06-20 11:34:07 on dist-test-slave-ctbw"
server_key: "5f4b5ec36de41bbfee2426668151f925"
server_key_iv: "1d4d4f1752379c1b7769ae10f96f5784"
server_key_version: "encryptionkey@0"
I20260620 11:34:07.466360 12328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "7c8092641bd54eea84d8aa80751bf390"
format_stamp: "Formatted at 2026-06-20 11:34:07 on dist-test-slave-ctbw"
server_key: "5f4b5ec36de41bbfee2426668151f925"
server_key_iv: "1d4d4f1752379c1b7769ae10f96f5784"
server_key_version: "encryptionkey@0"
I20260620 11:34:07.467741 12328 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:34:07.468508 12342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:07.468677 12328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:34:07.468781 12328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "7c8092641bd54eea84d8aa80751bf390"
format_stamp: "Formatted at 2026-06-20 11:34:07 on dist-test-slave-ctbw"
server_key: "5f4b5ec36de41bbfee2426668151f925"
server_key_iv: "1d4d4f1752379c1b7769ae10f96f5784"
server_key_version: "encryptionkey@0"
I20260620 11:34:07.468858 12328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:34:07.479792 12328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:34:07.480095 12328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:34:07.480248 12328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:34:07.480508 12328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:34:07.480856 12328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:34:07.480890 12328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:07.480912 12328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:34:07.480952 12328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:07.487985 12328 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:35649
I20260620 11:34:07.488013 12455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:35649 every 8 connection(s)
I20260620 11:34:07.488346 12328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:34:07.494496 12309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:07.489069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:43933 (local address 127.2.243.190:43919)
0620 11:34:07.489137 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:07.489140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:07.489464 (+ 324us) server_negotiation.cc:407] Connection header received
0620 11:34:07.490248 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:07.490249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:07.490262 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:07.490317 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:07.491290 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:07.491781 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:07.493095 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:07.493267 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:07.493317 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:07.494257 (+ 940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:34:07.494258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:34:07.494258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:34:07.494264 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:34:07.494277 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:34:07.494281 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:34:07.494323 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:07.494414 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 11:34:07.494437 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 11:34:07.494699 12458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:07.489067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43919 (local address 127.2.243.129:43933)
0620 11:34:07.489286 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:07.489329 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:07.490121 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:07.490461 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:07.490467 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:07.490480 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:07.491198 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:07.491204 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:07.491886 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:07.491887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:07.493007 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:07.493010 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:07.493358 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:07.493359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:07.493519 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:07.494111 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:34:07.494112 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:07.494153 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:34:07.494177 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:34:07.494342 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:34:07.494346 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:34:07.494348 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:34:07.494407 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 11:34:07.494487 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":85,"threads_started":1}
I20260620 11:34:07.495198 12456 heartbeater.cc:347] Connected to a master server at 127.2.243.190:43919
I20260620 11:34:07.495278 12456 heartbeater.cc:464] Registering TS with master...
I20260620 11:34:07.495464 12456 heartbeater.cc:511] Master 127.2.243.190:43919 requested a full tablet report, sending...
I20260620 11:34:07.496033 12268 ts_manager.cc:194] Registered new tserver with Master: 7c8092641bd54eea84d8aa80751bf390 (127.2.243.129:35649)
I20260620 11:34:07.496577 12268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.243.129:43933
I20260620 11:34:07.498095 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 12328
I20260620 11:34:07.498202 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:34:07.498412 3022 external_mini_cluster.cc:1428] Setting key 756174e947ce3195c40e0c4cab7bd30f
I20260620 11:34:07.499322 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:43919
--builtin_ntp_servers=127.2.243.148:32951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 11:34:07.579298 12459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:34:07.579546 12459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:34:07.579581 12459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:34:07.580734 12459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:34:07.580806 12459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:34:07.580849 12459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:34:07.582248 12459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:32951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:43919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:34:07.582544 12459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:34:07.582923 12459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:34:07.588023 12465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:07.588030 12467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:07.588034 12464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:34:07.588274 12459 server_base.cc:1034] running on GCE node
I20260620 11:34:07.588476 12459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:34:07.588685 12459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:34:07.589828 12459 hybrid_clock.cc:648] HybridClock initialized: now 1781955247589808 us; error 34 us; skew 500 ppm
I20260620 11:34:07.591228 12459 webserver.cc:460] Webserver started at http://127.2.243.130:40239/ using document root <none> and password file <none>
I20260620 11:34:07.591521 12459 fs_manager.cc:359] Metadata directory not provided
I20260620 11:34:07.591572 12459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:34:07.591750 12459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:34:07.592835 12459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "c41be59780f343e9bfd71f9e6a3b53c4"
format_stamp: "Formatted at 2026-06-20 11:34:07 on dist-test-slave-ctbw"
server_key: "09afd843bc8e27724565aebef5512385"
server_key_iv: "41a231f876f47cc798c6722a60f14da4"
server_key_version: "encryptionkey@0"
I20260620 11:34:07.593189 12459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "c41be59780f343e9bfd71f9e6a3b53c4"
format_stamp: "Formatted at 2026-06-20 11:34:07 on dist-test-slave-ctbw"
server_key: "09afd843bc8e27724565aebef5512385"
server_key_iv: "41a231f876f47cc798c6722a60f14da4"
server_key_version: "encryptionkey@0"
I20260620 11:34:07.594633 12459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:34:07.595561 12473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:07.595747 12459 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:34:07.595856 12459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "c41be59780f343e9bfd71f9e6a3b53c4"
format_stamp: "Formatted at 2026-06-20 11:34:07 on dist-test-slave-ctbw"
server_key: "09afd843bc8e27724565aebef5512385"
server_key_iv: "41a231f876f47cc798c6722a60f14da4"
server_key_version: "encryptionkey@0"
I20260620 11:34:07.595949 12459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:34:07.612191 12459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:34:07.612557 12459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:34:07.612731 12459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:34:07.612983 12459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:34:07.613390 12459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:34:07.613451 12459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:07.613497 12459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:34:07.613546 12459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:07.619856 12459 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:39653
I20260620 11:34:07.619917 12586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:39653 every 8 connection(s)
I20260620 11:34:07.620262 12459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:34:07.624171 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 12459
I20260620 11:34:07.624274 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:34:07.624521 3022 external_mini_cluster.cc:1428] Setting key 2385f26996a40d586f4f8494df7b09af
I20260620 11:34:07.625487 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:43919
--builtin_ntp_servers=127.2.243.148:32951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 11:34:07.626157 12309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:07.621083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:43695 (local address 127.2.243.190:43919)
0620 11:34:07.621170 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:07.621172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:07.621482 (+ 310us) server_negotiation.cc:407] Connection header received
0620 11:34:07.622033 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:07.622034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:07.622046 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:07.622119 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:07.622835 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:07.623370 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:07.624441 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:07.624741 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:07.624883 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:07.625880 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:34:07.625881 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:34:07.625882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:34:07.625886 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:34:07.625902 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:34:07.625906 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:34:07.625949 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:07.626050 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 11:34:07.626078 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 11:34:07.626405 12589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:07.621065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43919 (local address 127.2.243.130:43695)
0620 11:34:07.621341 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:07.621377 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:07.621913 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:07.622197 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:07.622201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:07.622209 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:07.622716 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:07.622720 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:07.623463 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:07.623465 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:07.624316 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:07.624319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:07.624907 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:07.624908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:07.625028 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:07.625722 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:34:07.625722 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:07.625751 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:34:07.625775 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:34:07.625960 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:34:07.625965 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:34:07.625968 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:34:07.626055 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 11:34:07.626167 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":104,"threads_started":1}
I20260620 11:34:07.626935 12587 heartbeater.cc:347] Connected to a master server at 127.2.243.190:43919
I20260620 11:34:07.627027 12587 heartbeater.cc:464] Registering TS with master...
I20260620 11:34:07.627223 12587 heartbeater.cc:511] Master 127.2.243.190:43919 requested a full tablet report, sending...
I20260620 11:34:07.627666 12268 ts_manager.cc:194] Registered new tserver with Master: c41be59780f343e9bfd71f9e6a3b53c4 (127.2.243.130:39653)
I20260620 11:34:07.628049 12268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.243.130:43695
W20260620 11:34:07.709290 12590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:34:07.709545 12590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:34:07.709569 12590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:34:07.710808 12590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:34:07.710857 12590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:34:07.710883 12590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:34:07.712324 12590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:32951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:43919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:34:07.712559 12590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:34:07.712817 12590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:34:07.718055 12595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:07.718031 12596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:07.718031 12598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:34:07.718117 12590 server_base.cc:1034] running on GCE node
I20260620 11:34:07.718467 12590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:34:07.718689 12590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:34:07.719849 12590 hybrid_clock.cc:648] HybridClock initialized: now 1781955247719827 us; error 36 us; skew 500 ppm
I20260620 11:34:07.721592 12590 webserver.cc:460] Webserver started at http://127.2.243.131:37967/ using document root <none> and password file <none>
I20260620 11:34:07.721858 12590 fs_manager.cc:359] Metadata directory not provided
I20260620 11:34:07.721907 12590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:34:07.722052 12590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:34:07.723230 12590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "f751930fd26746df8579654056e1aae3"
format_stamp: "Formatted at 2026-06-20 11:34:07 on dist-test-slave-ctbw"
server_key: "c40c904b39c0944850da96cfae4f8cd6"
server_key_iv: "3b9b5e914219c3ca23dbd78b03df96b6"
server_key_version: "encryptionkey@0"
I20260620 11:34:07.723600 12590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "f751930fd26746df8579654056e1aae3"
format_stamp: "Formatted at 2026-06-20 11:34:07 on dist-test-slave-ctbw"
server_key: "c40c904b39c0944850da96cfae4f8cd6"
server_key_iv: "3b9b5e914219c3ca23dbd78b03df96b6"
server_key_version: "encryptionkey@0"
I20260620 11:34:07.724939 12590 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:34:07.725782 12604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:07.725975 12590 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:34:07.726089 12590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "f751930fd26746df8579654056e1aae3"
format_stamp: "Formatted at 2026-06-20 11:34:07 on dist-test-slave-ctbw"
server_key: "c40c904b39c0944850da96cfae4f8cd6"
server_key_iv: "3b9b5e914219c3ca23dbd78b03df96b6"
server_key_version: "encryptionkey@0"
I20260620 11:34:07.726168 12590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:34:07.739744 12590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:34:07.740074 12590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:34:07.740232 12590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:34:07.740471 12590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:34:07.740808 12590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:34:07.740864 12590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:07.740898 12590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:34:07.740955 12590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:07.747020 12590 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:45463
I20260620 11:34:07.747092 12717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:45463 every 8 connection(s)
I20260620 11:34:07.747459 12590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:34:07.751314 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 12590
I20260620 11:34:07.751421 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:34:07.751619 3022 external_mini_cluster.cc:1428] Setting key ee26ba6113eabe627af0bce58465a6fc
I20260620 11:34:07.753783 12309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:07.748264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:38939 (local address 127.2.243.190:43919)
0620 11:34:07.748379 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:07.748382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:07.748636 (+ 254us) server_negotiation.cc:407] Connection header received
0620 11:34:07.749248 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:07.749250 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:07.749263 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:07.749316 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:07.750094 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:07.750662 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:07.752073 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:07.752451 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:07.752529 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:07.753487 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:34:07.753489 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:34:07.753489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:34:07.753494 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:34:07.753512 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:34:07.753516 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:34:07.753577 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:07.753664 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 11:34:07.753697 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 11:34:07.754062 12720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:07.748218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43919 (local address 127.2.243.131:38939)
0620 11:34:07.748483 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:07.748537 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:07.749122 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:07.749398 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:07.749409 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:07.749420 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:07.749989 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:07.749994 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:07.750770 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:07.750771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:07.751984 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:07.751986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:07.752548 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:07.752549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:07.752707 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:07.753310 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:34:07.753310 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:07.753342 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:34:07.753368 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:34:07.753584 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:34:07.753591 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:34:07.753595 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:34:07.753670 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 11:34:07.753783 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":89,"threads_started":1}
I20260620 11:34:07.754602 12718 heartbeater.cc:347] Connected to a master server at 127.2.243.190:43919
I20260620 11:34:07.754693 12718 heartbeater.cc:464] Registering TS with master...
I20260620 11:34:07.754969 12718 heartbeater.cc:511] Master 127.2.243.190:43919 requested a full tablet report, sending...
I20260620 11:34:07.755335 12268 ts_manager.cc:194] Registered new tserver with Master: f751930fd26746df8579654056e1aae3 (127.2.243.131:45463)
I20260620 11:34:07.755731 12268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.243.131:38939
I20260620 11:34:07.763703 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 11:34:07.765734 12309 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:07.765167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33054 (local address 127.2.243.190:43919)
0620 11:34:07.765231 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:07.765233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:07.765355 (+ 122us) server_negotiation.cc:407] Connection header received
0620 11:34:07.765438 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:07.765438 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:07.765451 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:07.765497 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:07.765635 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33054: BlockingRecv error: recv got EOF from 127.0.0.1:33054 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 11:34:07.765802 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 12328
I20260620 11:34:07.771018 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 12459
I20260620 11:34:07.775378 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 12590
I20260620 11:34:07.779623 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 12237
2026-06-20T11:34:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (598 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T11:34:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:34:07Z Disabled control of system clock
I20260620 11:34:07.806632 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:36987
--webserver_interface=127.2.243.190
--webserver_port=0
--builtin_ntp_servers=127.2.243.148:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:36987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 11:34:07.889263 12729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:34:07.889606 12729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:34:07.889660 12729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:34:07.890966 12729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:34:07.891033 12729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:34:07.891057 12729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:34:07.891088 12729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:34:07.891109 12729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:34:07.893548 12729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:36987
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:36987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:34:07.893823 12729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:34:07.894227 12729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:34:07.900436 12735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:07.900645 12734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:34:07.900902 12729 server_base.cc:1034] running on GCE node
W20260620 11:34:07.901019 12737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:34:07.901276 12729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:34:07.901530 12729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:34:07.902712 12729 hybrid_clock.cc:648] HybridClock initialized: now 1781955247902699 us; error 42 us; skew 500 ppm
I20260620 11:34:07.903998 12729 webserver.cc:460] Webserver started at http://127.2.243.190:37337/ using document root <none> and password file <none>
I20260620 11:34:07.904268 12729 fs_manager.cc:359] Metadata directory not provided
I20260620 11:34:07.904338 12729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:34:07.904482 12729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:34:07.905534 12729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "6e5f1c8360794d55a115978c9f16d8a7"
format_stamp: "Formatted at 2026-06-20 11:34:07 on dist-test-slave-ctbw"
server_key: "cb19f87af3b3fcbebf552abe721bb141"
server_key_iv: "f7024367f0d5ea1a6d219e43d5e77875"
server_key_version: "encryptionkey@0"
I20260620 11:34:07.905910 12729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "6e5f1c8360794d55a115978c9f16d8a7"
format_stamp: "Formatted at 2026-06-20 11:34:07 on dist-test-slave-ctbw"
server_key: "cb19f87af3b3fcbebf552abe721bb141"
server_key_iv: "f7024367f0d5ea1a6d219e43d5e77875"
server_key_version: "encryptionkey@0"
I20260620 11:34:07.907346 12729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:34:07.908150 12743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:07.908329 12729 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:34:07.908430 12729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "6e5f1c8360794d55a115978c9f16d8a7"
format_stamp: "Formatted at 2026-06-20 11:34:07 on dist-test-slave-ctbw"
server_key: "cb19f87af3b3fcbebf552abe721bb141"
server_key_iv: "f7024367f0d5ea1a6d219e43d5e77875"
server_key_version: "encryptionkey@0"
I20260620 11:34:07.908514 12729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:34:07.911917 12729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:34:07.912195 12729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:34:07.912339 12729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:34:07.916893 12729 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:36987
I20260620 11:34:07.916909 12795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:36987 every 8 connection(s)
I20260620 11:34:07.917384 12729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:34:07.918358 12796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:34:07.920719 12796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7: Bootstrap starting.
I20260620 11:34:07.920979 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 12729
I20260620 11:34:07.921108 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:34:07.921363 3022 external_mini_cluster.cc:1428] Setting key e133d250d999d694957f009458319b6b
I20260620 11:34:07.921381 12796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7: Neither blocks nor log segments found. Creating new log.
I20260620 11:34:07.921669 12796 log.cc:824] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7: Log is configured to *not* fsync() on all Append() calls
I20260620 11:34:07.922422 12796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7: No bootstrap required, opened a new log
I20260620 11:34:07.924075 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:07.921902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45142 (local address 127.2.243.190:36987)
0620 11:34:07.922114 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:07.922124 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:07.922159 (+ 35us) server_negotiation.cc:407] Connection header received
0620 11:34:07.922648 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:07.922656 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:07.922695 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:07.922900 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:07.923650 (+ 750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:34:07.923654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:34:07.923655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:34:07.923661 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:34:07.923708 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:34:07.923718 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:34:07.923813 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:07.923850 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 11:34:07.923923 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260620 11:34:07.924239 12796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5f1c8360794d55a115978c9f16d8a7" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36987 } }
I20260620 11:34:07.924412 12796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:34:07.924449 12796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e5f1c8360794d55a115978c9f16d8a7, State: Initialized, Role: FOLLOWER
I20260620 11:34:07.924567 12796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5f1c8360794d55a115978c9f16d8a7" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36987 } }
I20260620 11:34:07.924661 12796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:34:07.924693 12796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:34:07.924743 12796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:34:07.925478 12796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5f1c8360794d55a115978c9f16d8a7" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36987 } }
I20260620 11:34:07.925575 12796 leader_election.cc:302] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e5f1c8360794d55a115978c9f16d8a7; no voters:
I20260620 11:34:07.925698 12796 leader_election.cc:288] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:34:07.925768 12801 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:34:07.925947 12801 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7 [term 1 LEADER]: Becoming Leader. State: Replica: 6e5f1c8360794d55a115978c9f16d8a7, State: Running, Role: LEADER
I20260620 11:34:07.925979 12796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:34:07.926047 12801 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5f1c8360794d55a115978c9f16d8a7" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36987 } }
I20260620 11:34:07.926470 12802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e5f1c8360794d55a115978c9f16d8a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5f1c8360794d55a115978c9f16d8a7" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36987 } } }
I20260620 11:34:07.926614 12802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7 [sys.catalog]: This master's current role is: LEADER
I20260620 11:34:07.926466 12803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e5f1c8360794d55a115978c9f16d8a7. Latest consensus state: current_term: 1 leader_uuid: "6e5f1c8360794d55a115978c9f16d8a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e5f1c8360794d55a115978c9f16d8a7" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36987 } } }
I20260620 11:34:07.926967 12803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7 [sys.catalog]: This master's current role is: LEADER
I20260620 11:34:07.926971 12810 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:34:07.927397 12810 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:34:07.928859 12810 catalog_manager.cc:1347] Generated new cluster ID: 74b29783554a4794992d8c385a6f5237
I20260620 11:34:07.928922 12810 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:34:07.950910 12810 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:34:07.951114 12810 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:34:07.966998 12810 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e5f1c8360794d55a115978c9f16d8a7: Generated new TSK 0
I20260620 11:34:07.967235 12810 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:34:07.978811 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:36987
--builtin_ntp_servers=127.2.243.148:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 11:34:08.064956 12820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:34:08.065191 12820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:34:08.065215 12820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:34:08.066355 12820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:34:08.066392 12820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:34:08.066425 12820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:34:08.067828 12820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:36987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:34:08.068046 12820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:34:08.068320 12820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:34:08.074043 12825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:08.074054 12826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:34:08.074148 12820 server_base.cc:1034] running on GCE node
W20260620 11:34:08.074111 12828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:34:08.074491 12820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:34:08.074739 12820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:34:08.075955 12820 hybrid_clock.cc:648] HybridClock initialized: now 1781955248075951 us; error 40 us; skew 500 ppm
I20260620 11:34:08.077297 12820 webserver.cc:460] Webserver started at http://127.2.243.129:37785/ using document root <none> and password file <none>
I20260620 11:34:08.077558 12820 fs_manager.cc:359] Metadata directory not provided
I20260620 11:34:08.077631 12820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:34:08.077790 12820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:34:08.079018 12820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "2603eec9d2944f7c8b7fc3d4f6c8faa2"
format_stamp: "Formatted at 2026-06-20 11:34:08 on dist-test-slave-ctbw"
server_key: "8e42a1c37caf55bb9b49b9c9fd395bbc"
server_key_iv: "6f14099458fbb528e9b0c8c806ada3a7"
server_key_version: "encryptionkey@0"
I20260620 11:34:08.079406 12820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "2603eec9d2944f7c8b7fc3d4f6c8faa2"
format_stamp: "Formatted at 2026-06-20 11:34:08 on dist-test-slave-ctbw"
server_key: "8e42a1c37caf55bb9b49b9c9fd395bbc"
server_key_iv: "6f14099458fbb528e9b0c8c806ada3a7"
server_key_version: "encryptionkey@0"
I20260620 11:34:08.080799 12820 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:34:08.081553 12834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:08.081738 12820 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:34:08.081845 12820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "2603eec9d2944f7c8b7fc3d4f6c8faa2"
format_stamp: "Formatted at 2026-06-20 11:34:08 on dist-test-slave-ctbw"
server_key: "8e42a1c37caf55bb9b49b9c9fd395bbc"
server_key_iv: "6f14099458fbb528e9b0c8c806ada3a7"
server_key_version: "encryptionkey@0"
I20260620 11:34:08.081935 12820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:34:08.085357 12820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:34:08.085623 12820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:34:08.085763 12820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:34:08.085963 12820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:34:08.086289 12820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:34:08.086335 12820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:08.086373 12820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:34:08.086401 12820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:08.092152 12820 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:44633
I20260620 11:34:08.092193 12947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:44633 every 8 connection(s)
I20260620 11:34:08.092547 12820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:34:08.094173 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 12820
I20260620 11:34:08.094281 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:34:08.094523 3022 external_mini_cluster.cc:1428] Setting key a4688be956857f91b16393e3d7137196
I20260620 11:34:08.095669 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:36987
--builtin_ntp_servers=127.2.243.148:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 11:34:08.096289 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:08.093346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:47189 (local address 127.2.243.190:36987)
0620 11:34:08.093407 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:08.093408 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:08.093664 (+ 256us) server_negotiation.cc:407] Connection header received
0620 11:34:08.094347 (+ 683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:08.094349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:08.094356 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:08.094432 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:08.096050 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:34:08.096052 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:34:08.096052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:34:08.096060 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:34:08.096079 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:34:08.096083 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:34:08.096138 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:08.096227 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 11:34:08.096238 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 11:34:08.096560 12950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:08.093340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:36987 (local address 127.2.243.129:47189)
0620 11:34:08.093539 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:08.093574 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:08.094189 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:08.094565 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:08.094570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:08.094582 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:08.095858 (+ 1276us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:34:08.095858 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:08.095898 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:34:08.095934 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:34:08.096149 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:34:08.096154 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:34:08.096157 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:34:08.096238 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 11:34:08.096306 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":95,"threads_started":1}
I20260620 11:34:08.097056 12948 heartbeater.cc:347] Connected to a master server at 127.2.243.190:36987
I20260620 11:34:08.097169 12948 heartbeater.cc:464] Registering TS with master...
I20260620 11:34:08.097323 12948 heartbeater.cc:511] Master 127.2.243.190:36987 requested a full tablet report, sending...
I20260620 11:34:08.097843 12760 ts_manager.cc:194] Registered new tserver with Master: 2603eec9d2944f7c8b7fc3d4f6c8faa2 (127.2.243.129:44633)
W20260620 11:34:08.179091 12951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:34:08.179311 12951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:34:08.179333 12951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:34:08.180646 12951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:34:08.180691 12951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:34:08.180718 12951 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:34:08.182094 12951 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:36987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12951
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:34:08.182368 12951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:34:08.182688 12951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 11:34:08.187975 12951 server_base.cc:1034] running on GCE node
W20260620 11:34:08.188027 12956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:08.187951 12959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:08.187958 12957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:34:08.188354 12951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:34:08.188580 12951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:34:08.189730 12951 hybrid_clock.cc:648] HybridClock initialized: now 1781955248189702 us; error 46 us; skew 500 ppm
I20260620 11:34:08.191051 12951 webserver.cc:460] Webserver started at http://127.2.243.130:36555/ using document root <none> and password file <none>
I20260620 11:34:08.191264 12951 fs_manager.cc:359] Metadata directory not provided
I20260620 11:34:08.191306 12951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:34:08.191397 12951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:34:08.192418 12951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "527d9d5da48a4ff78b69ed291a488ffc"
format_stamp: "Formatted at 2026-06-20 11:34:08 on dist-test-slave-ctbw"
server_key: "b0f8df90e74c7d877313e87a5b5871e5"
server_key_iv: "38d093ae651f72faf403490f5a0aa27e"
server_key_version: "encryptionkey@0"
I20260620 11:34:08.192768 12951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "527d9d5da48a4ff78b69ed291a488ffc"
format_stamp: "Formatted at 2026-06-20 11:34:08 on dist-test-slave-ctbw"
server_key: "b0f8df90e74c7d877313e87a5b5871e5"
server_key_iv: "38d093ae651f72faf403490f5a0aa27e"
server_key_version: "encryptionkey@0"
I20260620 11:34:08.194048 12951 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:34:08.194734 12965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:08.194914 12951 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:34:08.195005 12951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "527d9d5da48a4ff78b69ed291a488ffc"
format_stamp: "Formatted at 2026-06-20 11:34:08 on dist-test-slave-ctbw"
server_key: "b0f8df90e74c7d877313e87a5b5871e5"
server_key_iv: "38d093ae651f72faf403490f5a0aa27e"
server_key_version: "encryptionkey@0"
I20260620 11:34:08.195062 12951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:34:08.198387 12951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:34:08.198609 12951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:34:08.198763 12951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:34:08.198992 12951 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:34:08.199252 12951 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:34:08.199282 12951 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:08.199302 12951 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:34:08.199316 12951 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:08.205358 12951 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:36119
I20260620 11:34:08.205513 13078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:36119 every 8 connection(s)
I20260620 11:34:08.205807 12951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:34:08.209178 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:08.206619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:40223 (local address 127.2.243.190:36987)
0620 11:34:08.206680 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:08.206682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:08.206969 (+ 287us) server_negotiation.cc:407] Connection header received
0620 11:34:08.207507 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:08.207508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:08.207515 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:08.207568 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:08.208887 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:34:08.208889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:34:08.208890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:34:08.208894 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:34:08.208919 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:34:08.208923 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:34:08.208995 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:08.209089 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 11:34:08.209104 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 11:34:08.209421 13081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:08.206620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:36987 (local address 127.2.243.130:40223)
0620 11:34:08.206789 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:08.206823 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:08.207380 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:08.207645 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:08.207648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:08.207656 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:08.208716 (+ 1060us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:34:08.208717 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:08.208747 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:34:08.208775 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:34:08.208994 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:34:08.209000 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:34:08.209003 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:34:08.209102 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 11:34:08.209165 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260620 11:34:08.209950 13079 heartbeater.cc:347] Connected to a master server at 127.2.243.190:36987
I20260620 11:34:08.210027 13079 heartbeater.cc:464] Registering TS with master...
I20260620 11:34:08.210193 13079 heartbeater.cc:511] Master 127.2.243.190:36987 requested a full tablet report, sending...
I20260620 11:34:08.210664 12760 ts_manager.cc:194] Registered new tserver with Master: 527d9d5da48a4ff78b69ed291a488ffc (127.2.243.130:36119)
I20260620 11:34:08.210672 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 12951
I20260620 11:34:08.210788 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:34:08.211074 3022 external_mini_cluster.cc:1428] Setting key 9ad2f5bacd6657ad5939c25071725bcf
I20260620 11:34:08.211942 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:36987
--builtin_ntp_servers=127.2.243.148:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 11:34:08.293303 13082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:34:08.293545 13082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:34:08.293569 13082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:34:08.294729 13082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:34:08.294766 13082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:34:08.294791 13082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:34:08.296625 13082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:36987
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:34:08.296825 13082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:34:08.297082 13082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:34:08.302289 13090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:08.302309 13088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:08.302284 13087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:34:08.302567 13082 server_base.cc:1034] running on GCE node
I20260620 11:34:08.302762 13082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:34:08.303023 13082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:34:08.304203 13082 hybrid_clock.cc:648] HybridClock initialized: now 1781955248304193 us; error 28 us; skew 500 ppm
I20260620 11:34:08.305594 13082 webserver.cc:460] Webserver started at http://127.2.243.131:38547/ using document root <none> and password file <none>
I20260620 11:34:08.305799 13082 fs_manager.cc:359] Metadata directory not provided
I20260620 11:34:08.305846 13082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:34:08.305975 13082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:34:08.307150 13082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "1dc50a1c8b7e46be9463b53e4620799d"
format_stamp: "Formatted at 2026-06-20 11:34:08 on dist-test-slave-ctbw"
server_key: "00937926082379b7051d0151e557a341"
server_key_iv: "1988785aa6f95ffc7e0b58047a0d0834"
server_key_version: "encryptionkey@0"
I20260620 11:34:08.307473 13082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "1dc50a1c8b7e46be9463b53e4620799d"
format_stamp: "Formatted at 2026-06-20 11:34:08 on dist-test-slave-ctbw"
server_key: "00937926082379b7051d0151e557a341"
server_key_iv: "1988785aa6f95ffc7e0b58047a0d0834"
server_key_version: "encryptionkey@0"
I20260620 11:34:08.308804 13082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 11:34:08.309528 13096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:08.309712 13082 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:34:08.309787 13082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "1dc50a1c8b7e46be9463b53e4620799d"
format_stamp: "Formatted at 2026-06-20 11:34:08 on dist-test-slave-ctbw"
server_key: "00937926082379b7051d0151e557a341"
server_key_iv: "1988785aa6f95ffc7e0b58047a0d0834"
server_key_version: "encryptionkey@0"
I20260620 11:34:08.309840 13082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:34:08.313503 13082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:34:08.313722 13082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:34:08.313812 13082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:34:08.313966 13082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:34:08.314201 13082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:34:08.314230 13082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:08.314250 13082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:34:08.314263 13082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:08.320371 13082 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:36547
I20260620 11:34:08.320431 13209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:36547 every 8 connection(s)
I20260620 11:34:08.320748 13082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:34:08.323819 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:08.321482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:36475 (local address 127.2.243.190:36987)
0620 11:34:08.321550 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:08.321552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:08.321791 (+ 239us) server_negotiation.cc:407] Connection header received
0620 11:34:08.322290 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:08.322291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:08.322297 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:08.322346 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:08.323555 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:34:08.323556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:34:08.323556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:34:08.323561 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:34:08.323576 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:34:08.323580 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:34:08.323633 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:08.323730 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 11:34:08.323746 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 11:34:08.324043 13212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:08.321483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:36987 (local address 127.2.243.131:36475)
0620 11:34:08.321671 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:08.321704 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:08.322186 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:08.322421 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:08.322424 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:08.322432 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:08.323383 (+ 951us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:34:08.323383 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:08.323413 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:34:08.323442 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:34:08.323642 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:34:08.323647 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:34:08.323649 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:34:08.323740 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 11:34:08.323802 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260620 11:34:08.324577 13210 heartbeater.cc:347] Connected to a master server at 127.2.243.190:36987
I20260620 11:34:08.324656 13210 heartbeater.cc:464] Registering TS with master...
I20260620 11:34:08.324819 13210 heartbeater.cc:511] Master 127.2.243.190:36987 requested a full tablet report, sending...
I20260620 11:34:08.325209 12760 ts_manager.cc:194] Registered new tserver with Master: 1dc50a1c8b7e46be9463b53e4620799d (127.2.243.131:36547)
I20260620 11:34:08.326519 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 13082
I20260620 11:34:08.326617 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:34:08.326826 3022 external_mini_cluster.cc:1428] Setting key 2ab9530c2209539d2f372b7bcf7d896b
I20260620 11:34:08.328024 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 11:34:08.329924 12800 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:08.329352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.2.243.190:36987)
0620 11:34:08.329418 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:08.329420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:08.329597 (+ 177us) server_negotiation.cc:407] Connection header received
0620 11:34:08.329674 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:08.329675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:08.329680 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:08.329737 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:08.329860 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45154: BlockingRecv error: recv got EOF from 127.0.0.1:45154 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 11:34:08.330014 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 12820
I20260620 11:34:08.334986 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 12951
I20260620 11:34:08.339357 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 13082
I20260620 11:34:08.343339 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 12729
2026-06-20T11:34:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (563 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:34:08 dist-test-slave-ctbw krb5kdc[13220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:34:08 dist-test-slave-ctbw krb5kdc[13220](info): set up 2 sockets
Jun 20 11:34:08 dist-test-slave-ctbw krb5kdc[13220](info): commencing operation
krb5kdc: starting...
W20260620 11:34:09.392283 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:34:09 dist-test-slave-ctbw krb5kdc[13220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:34:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:34:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:34:09.510262 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:35141
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:35141
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:34:09.606104 13233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:34:09.606338 13233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:34:09.606364 13233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:34:09.607677 13233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:34:09.607729 13233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:34:09.607746 13233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:34:09.607760 13233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:34:09.607816 13233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:34:09.609346 13233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:35141
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:35141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:34:09.609630 13233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:34:09.609890 13233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:34:09.614914 13239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:34:09.614989 13233 server_base.cc:1034] running on GCE node
W20260620 11:34:09.614928 13241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:09.614929 13238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:34:09.615355 13233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:34:09.615662 13233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:34:09.616817 13233 hybrid_clock.cc:648] HybridClock initialized: now 1781955249616794 us; error 38 us; skew 500 ppm
Jun 20 11:34:09 dist-test-slave-ctbw krb5kdc[13220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955249, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:34:09.619527 13233 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:34:09.620033 13233 webserver.cc:460] Webserver started at http://127.2.243.190:37103/ using document root <none> and password file <none>
I20260620 11:34:09.620296 13233 fs_manager.cc:359] Metadata directory not provided
I20260620 11:34:09.620343 13233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:34:09.620479 13233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:34:09.621613 13233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "8bd8f12e190245baa0a49851bf79d29d"
format_stamp: "Formatted at 2026-06-20 11:34:09 on dist-test-slave-ctbw"
server_key: "d53ae149c29ef836b3275994e3080c5a"
server_key_iv: "d3e9f7b506f66f5e7d88f0fa5343a15d"
server_key_version: "encryptionkey@0"
I20260620 11:34:09.621963 13233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "8bd8f12e190245baa0a49851bf79d29d"
format_stamp: "Formatted at 2026-06-20 11:34:09 on dist-test-slave-ctbw"
server_key: "d53ae149c29ef836b3275994e3080c5a"
server_key_iv: "d3e9f7b506f66f5e7d88f0fa5343a15d"
server_key_version: "encryptionkey@0"
I20260620 11:34:09.623409 13233 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:34:09.624186 13248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:09.624433 13233 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:34:09.624532 13233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "8bd8f12e190245baa0a49851bf79d29d"
format_stamp: "Formatted at 2026-06-20 11:34:09 on dist-test-slave-ctbw"
server_key: "d53ae149c29ef836b3275994e3080c5a"
server_key_iv: "d3e9f7b506f66f5e7d88f0fa5343a15d"
server_key_version: "encryptionkey@0"
I20260620 11:34:09.624612 13233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:34:09.653560 13233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:34:09.655997 13233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:34:09.656176 13233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:34:09.660547 13233 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:35141
I20260620 11:34:09.660580 13300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:35141 every 8 connection(s)
I20260620 11:34:09.661002 13233 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:34:09.662029 13301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:34:09.664440 13301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d: Bootstrap starting.
I20260620 11:34:09.665024 13301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d: Neither blocks nor log segments found. Creating new log.
I20260620 11:34:09.665304 13301 log.cc:824] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d: Log is configured to *not* fsync() on all Append() calls
I20260620 11:34:09.665628 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 13233
I20260620 11:34:09.665710 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:34:09.665920 3022 external_mini_cluster.cc:1428] Setting key ff10cb63e8b4d21c990d73bec9222670
I20260620 11:34:09.666193 13301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d: No bootstrap required, opened a new log
I20260620 11:34:09.668043 13301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd8f12e190245baa0a49851bf79d29d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 35141 } }
I20260620 11:34:09.668224 13301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:34:09.668249 13301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bd8f12e190245baa0a49851bf79d29d, State: Initialized, Role: FOLLOWER
I20260620 11:34:09.668432 13301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd8f12e190245baa0a49851bf79d29d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 35141 } }
I20260620 11:34:09.668524 13301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:34:09.668555 13301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:34:09.668609 13301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:34:09.669660 13301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd8f12e190245baa0a49851bf79d29d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 35141 } }
I20260620 11:34:09.669749 13301 leader_election.cc:302] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8bd8f12e190245baa0a49851bf79d29d; no voters:
I20260620 11:34:09.669919 13301 leader_election.cc:288] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:34:09.669960 13306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:34:09.670106 13306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d [term 1 LEADER]: Becoming Leader. State: Replica: 8bd8f12e190245baa0a49851bf79d29d, State: Running, Role: LEADER
I20260620 11:34:09.670194 13301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:34:09.670188 13306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd8f12e190245baa0a49851bf79d29d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 35141 } }
I20260620 11:34:09.670843 13308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bd8f12e190245baa0a49851bf79d29d. Latest consensus state: current_term: 1 leader_uuid: "8bd8f12e190245baa0a49851bf79d29d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd8f12e190245baa0a49851bf79d29d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 35141 } } }
I20260620 11:34:09.670856 13307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bd8f12e190245baa0a49851bf79d29d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8bd8f12e190245baa0a49851bf79d29d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 35141 } } }
I20260620 11:34:09.671031 13308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d [sys.catalog]: This master's current role is: LEADER
I20260620 11:34:09.671058 13307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d [sys.catalog]: This master's current role is: LEADER
I20260620 11:34:09.671537 13316 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:34:09.671998 13316 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:34:09.673615 13316 catalog_manager.cc:1347] Generated new cluster ID: 5d708e4ee52b4ba4b1ff458f835f6be6
I20260620 11:34:09.673666 13316 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:34:09 dist-test-slave-ctbw krb5kdc[13220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:34:09.679031 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:09.666468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42858 (local address 127.2.243.190:35141)
0620 11:34:09.666709 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:09.666718 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:09.666752 (+ 34us) server_negotiation.cc:407] Connection header received
0620 11:34:09.668066 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:09.668075 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:09.668135 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:09.668338 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:09.669126 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:09.671279 (+ 2153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:09.672382 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:09.673002 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:09.673104 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:09.675598 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:34:09.675602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:34:09.675603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:34:09.675615 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:34:09.677313 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:34:09.677715 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:34:09.677720 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:34:09.677720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:34:09.677829 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:34:09.678039 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:34:09.678046 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:34:09.678046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:34:09.678303 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:34:09.678489 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:09.678560 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 11:34:09.678704 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":147,"threads_started":1}
I20260620 11:34:09.688052 13316 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:34:09.688634 13316 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:34:09.696281 13316 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d: Generated new TSK 0
I20260620 11:34:09.696501 13316 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
I20260620 11:34:09.727959 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:35141
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:34:09.831681 13327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:34:09.831905 13327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:34:09.831931 13327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:34:09.833137 13327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:34:09.833185 13327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:34:09.833213 13327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:34:09.834745 13327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:35141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:34:09.835011 13327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:34:09.835289 13327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:34:09.840492 13332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:09.840497 13333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:09.840534 13335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:34:09.840914 13327 server_base.cc:1034] running on GCE node
I20260620 11:34:09.841125 13327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:34:09.841347 13327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:34:09.842485 13327 hybrid_clock.cc:648] HybridClock initialized: now 1781955249842473 us; error 30 us; skew 500 ppm
Jun 20 11:34:09 dist-test-slave-ctbw krb5kdc[13220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955249, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:34:09.845255 13327 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:34:09.845777 13327 webserver.cc:460] Webserver started at http://127.2.243.129:41541/ using document root <none> and password file <none>
I20260620 11:34:09.846022 13327 fs_manager.cc:359] Metadata directory not provided
I20260620 11:34:09.846093 13327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:34:09.846247 13327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:34:09.847406 13327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "39fd7cfff83c49bd94059cf347ae2bb1"
format_stamp: "Formatted at 2026-06-20 11:34:09 on dist-test-slave-ctbw"
server_key: "67f3974c2c7dab662096057d7337023e"
server_key_iv: "662dc80c1095a449d8b905cfa76e500c"
server_key_version: "encryptionkey@0"
I20260620 11:34:09.847785 13327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "39fd7cfff83c49bd94059cf347ae2bb1"
format_stamp: "Formatted at 2026-06-20 11:34:09 on dist-test-slave-ctbw"
server_key: "67f3974c2c7dab662096057d7337023e"
server_key_iv: "662dc80c1095a449d8b905cfa76e500c"
server_key_version: "encryptionkey@0"
I20260620 11:34:09.849143 13327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:34:09.849973 13342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:09.850201 13327 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:34:09.850323 13327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "39fd7cfff83c49bd94059cf347ae2bb1"
format_stamp: "Formatted at 2026-06-20 11:34:09 on dist-test-slave-ctbw"
server_key: "67f3974c2c7dab662096057d7337023e"
server_key_iv: "662dc80c1095a449d8b905cfa76e500c"
server_key_version: "encryptionkey@0"
I20260620 11:34:09.850402 13327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:34:09.875458 13327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:34:09.877566 13327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:34:09.877753 13327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:34:09.878003 13327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:34:09.878351 13327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:34:09.878409 13327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:09.878455 13327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:34:09.878480 13327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:09.884531 13327 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:45805
I20260620 11:34:09.884654 13455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:45805 every 8 connection(s)
I20260620 11:34:09.884953 13327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
Jun 20 11:34:09 dist-test-slave-ctbw krb5kdc[13220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955249, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:34:09.894404 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 13327
I20260620 11:34:09.894522 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:34:09.894757 3022 external_mini_cluster.cc:1428] Setting key 4dd9bd660657814c0abc2f57591d2814
I20260620 11:34:09.896921 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:09.885709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:34263 (local address 127.2.243.190:35141)
0620 11:34:09.885783 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:09.885790 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:09.885956 (+ 166us) server_negotiation.cc:407] Connection header received
0620 11:34:09.886469 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:09.886470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:09.886480 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:09.886544 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:09.887592 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:09.888114 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:09.889316 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:09.889707 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:09.889787 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:09.893032 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:34:09.893034 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:34:09.893035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:34:09.893049 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:34:09.895136 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:34:09.895691 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:34:09.895693 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:34:09.895694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:34:09.895759 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:34:09.896063 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:34:09.896064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:34:09.896065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:34:09.896231 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:34:09.896278 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:09.896671 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 11:34:09.896775 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 11:34:09.897490 13458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:09.885674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:35141 (local address 127.2.243.129:34263)
0620 11:34:09.885836 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:09.885879 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:09.886377 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:09.886598 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:09.886601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:09.887281 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:09.887497 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:09.887502 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:09.888234 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:09.888235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:09.889211 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:09.889214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:09.889827 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:09.889829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:09.890020 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:09.890700 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:09.890712 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:09.892890 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:34:09.895232 (+ 2342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:34:09.895243 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:34:09.895245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:34:09.895586 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:34:09.895838 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:34:09.895839 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:34:09.895839 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:34:09.895988 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:34:09.896324 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:34:09.896331 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:34:09.896519 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 11:34:09.897025 (+ 506us) client_negotiation.cc:240] Negotiation successful
0620 11:34:09.897251 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":91,"threads_started":1}
I20260620 11:34:09.898068 13456 heartbeater.cc:347] Connected to a master server at 127.2.243.190:35141
I20260620 11:34:09.898203 13456 heartbeater.cc:464] Registering TS with master...
I20260620 11:34:09.898403 13456 heartbeater.cc:511] Master 127.2.243.190:35141 requested a full tablet report, sending...
I20260620 11:34:09.899029 13265 ts_manager.cc:194] Registered new tserver with Master: 39fd7cfff83c49bd94059cf347ae2bb1 (127.2.243.129:45805)
I20260620 11:34:09.899925 13265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:34263
WARNING: no policy specified for kudu/127.2.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
I20260620 11:34:09.922637 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:35141
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:34:10.006161 13461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:34:10.006448 13461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:34:10.006487 13461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:34:10.007735 13461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:34:10.007812 13461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:34:10.007860 13461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:34:10.009349 13461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:35141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:34:10.009675 13461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:34:10.009940 13461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:34:10.015246 13467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:10.015254 13469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:10.015254 13466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:34:10.015499 13461 server_base.cc:1034] running on GCE node
I20260620 11:34:10.015741 13461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:34:10.016006 13461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:34:10.017153 13461 hybrid_clock.cc:648] HybridClock initialized: now 1781955250017143 us; error 40 us; skew 500 ppm
Jun 20 11:34:10 dist-test-slave-ctbw krb5kdc[13220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955250, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:34:10.019827 13461 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:34:10.020344 13461 webserver.cc:460] Webserver started at http://127.2.243.130:37509/ using document root <none> and password file <none>
I20260620 11:34:10.020614 13461 fs_manager.cc:359] Metadata directory not provided
I20260620 11:34:10.020689 13461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:34:10.020861 13461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:34:10.021973 13461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "29c6eb5e1100421d9f49a7fd9cb20455"
format_stamp: "Formatted at 2026-06-20 11:34:10 on dist-test-slave-ctbw"
server_key: "cea0754530e4fc8308be7f43567c4e42"
server_key_iv: "7fa47dd3405a58140e3ac3cdac22f091"
server_key_version: "encryptionkey@0"
I20260620 11:34:10.022353 13461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "29c6eb5e1100421d9f49a7fd9cb20455"
format_stamp: "Formatted at 2026-06-20 11:34:10 on dist-test-slave-ctbw"
server_key: "cea0754530e4fc8308be7f43567c4e42"
server_key_iv: "7fa47dd3405a58140e3ac3cdac22f091"
server_key_version: "encryptionkey@0"
I20260620 11:34:10.023861 13461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260620 11:34:10.024679 13476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:10.024870 13461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:34:10.024978 13461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "29c6eb5e1100421d9f49a7fd9cb20455"
format_stamp: "Formatted at 2026-06-20 11:34:10 on dist-test-slave-ctbw"
server_key: "cea0754530e4fc8308be7f43567c4e42"
server_key_iv: "7fa47dd3405a58140e3ac3cdac22f091"
server_key_version: "encryptionkey@0"
I20260620 11:34:10.025055 13461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:34:10.039832 13461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:34:10.042322 13461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:34:10.042497 13461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:34:10.042766 13461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:34:10.043172 13461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:34:10.043228 13461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:10.043272 13461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:34:10.043299 13461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:10.049577 13461 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:35085
I20260620 11:34:10.049629 13589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:35085 every 8 connection(s)
I20260620 11:34:10.049997 13461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
Jun 20 11:34:10 dist-test-slave-ctbw krb5kdc[13220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955250, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:34:10.058058 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 13461
I20260620 11:34:10.058189 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:34:10.058475 3022 external_mini_cluster.cc:1428] Setting key e48a5f6f1aced6a9229455697c566468
I20260620 11:34:10.060299 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:10.050769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:60573 (local address 127.2.243.190:35141)
0620 11:34:10.050819 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:10.050821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:10.051017 (+ 196us) server_negotiation.cc:407] Connection header received
0620 11:34:10.051538 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:10.051539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:10.051549 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:10.051615 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:10.052586 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.053087 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.054158 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.054439 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.054507 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:10.056775 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:34:10.056777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:34:10.056777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:34:10.056784 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:34:10.058492 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:34:10.059083 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:34:10.059084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:34:10.059085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:34:10.059160 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:34:10.059458 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:34:10.059459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:34:10.059460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:34:10.059634 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:34:10.059728 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:10.059999 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 11:34:10.060130 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:34:10.060386 13592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:10.050697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:35141 (local address 127.2.243.130:60573)
0620 11:34:10.050859 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:10.050921 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:10.051441 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:10.051695 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:10.051699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:10.052293 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:10.052489 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:10.052494 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.053193 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.053194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:10.054053 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:10.054055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.054521 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.054522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:10.054629 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:10.055214 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:10.055224 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:10.056670 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:34:10.058618 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:34:10.058625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:34:10.058627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:34:10.058999 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:34:10.059254 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:34:10.059255 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:34:10.059256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:34:10.059393 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:34:10.059711 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:34:10.059715 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:34:10.059842 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 11:34:10.059922 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 11:34:10.060050 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":86,"threads_started":1}
I20260620 11:34:10.060977 13590 heartbeater.cc:347] Connected to a master server at 127.2.243.190:35141
I20260620 11:34:10.061127 13590 heartbeater.cc:464] Registering TS with master...
I20260620 11:34:10.061352 13590 heartbeater.cc:511] Master 127.2.243.190:35141 requested a full tablet report, sending...
I20260620 11:34:10.061793 13265 ts_manager.cc:194] Registered new tserver with Master: 29c6eb5e1100421d9f49a7fd9cb20455 (127.2.243.130:35085)
I20260620 11:34:10.062343 13265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:60573
WARNING: no policy specified for kudu/127.2.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
I20260620 11:34:10.085935 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:35141
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:35509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:34:10.169133 13595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:34:10.169353 13595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:34:10.169377 13595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:34:10.170642 13595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:34:10.170722 13595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:34:10.170758 13595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:34:10.172428 13595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:35509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:35141
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:34:10.172619 13595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:34:10.172809 13595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:34:10.177915 13600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:10.177918 13601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:34:10.178117 13595 server_base.cc:1034] running on GCE node
W20260620 11:34:10.177947 13603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:34:10.178431 13595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:34:10.178746 13595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:34:10.179893 13595 hybrid_clock.cc:648] HybridClock initialized: now 1781955250179878 us; error 62 us; skew 500 ppm
Jun 20 11:34:10 dist-test-slave-ctbw krb5kdc[13220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955250, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:34:10.182519 13595 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:34:10.183068 13595 webserver.cc:460] Webserver started at http://127.2.243.131:39375/ using document root <none> and password file <none>
I20260620 11:34:10.183303 13595 fs_manager.cc:359] Metadata directory not provided
I20260620 11:34:10.183349 13595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:34:10.183494 13595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:34:10.184546 13595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "ec49a2c90ae04a588173de1706d52a6b"
format_stamp: "Formatted at 2026-06-20 11:34:10 on dist-test-slave-ctbw"
server_key: "994e9b7adc4fbdd3aef44c084628f61d"
server_key_iv: "191e11cc091fd918355ce296df593ca0"
server_key_version: "encryptionkey@0"
I20260620 11:34:10.184916 13595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "ec49a2c90ae04a588173de1706d52a6b"
format_stamp: "Formatted at 2026-06-20 11:34:10 on dist-test-slave-ctbw"
server_key: "994e9b7adc4fbdd3aef44c084628f61d"
server_key_iv: "191e11cc091fd918355ce296df593ca0"
server_key_version: "encryptionkey@0"
I20260620 11:34:10.186215 13595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:34:10.187114 13610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:10.187326 13595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:34:10.187435 13595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "ec49a2c90ae04a588173de1706d52a6b"
format_stamp: "Formatted at 2026-06-20 11:34:10 on dist-test-slave-ctbw"
server_key: "994e9b7adc4fbdd3aef44c084628f61d"
server_key_iv: "191e11cc091fd918355ce296df593ca0"
server_key_version: "encryptionkey@0"
I20260620 11:34:10.187512 13595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:34:10.206393 13595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:34:10.208829 13595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:34:10.208997 13595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:34:10.209242 13595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:34:10.209571 13595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:34:10.209627 13595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:10.209673 13595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:34:10.209712 13595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:10.215479 13595 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:41901
I20260620 11:34:10.215521 13723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:41901 every 8 connection(s)
I20260620 11:34:10.215889 13595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:34:10.220989 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 13595
I20260620 11:34:10.221088 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:34:10.221289 3022 external_mini_cluster.cc:1428] Setting key b364b150f66597f984de66226c02dc37
Jun 20 11:34:10 dist-test-slave-ctbw krb5kdc[13220](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955250, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:34:10.225780 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:10.216690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:59227 (local address 127.2.243.190:35141)
0620 11:34:10.216753 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:10.216757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:10.216899 (+ 142us) server_negotiation.cc:407] Connection header received
0620 11:34:10.217379 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:10.217380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:10.217388 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:10.217451 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:10.218348 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.218831 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.219869 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.220056 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.220112 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:10.222769 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:34:10.222771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:34:10.222772 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:34:10.222781 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:34:10.224177 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:34:10.224667 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:34:10.224669 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:34:10.224669 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:34:10.224736 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:34:10.225014 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:34:10.225015 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:34:10.225016 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:34:10.225178 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:34:10.225264 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:10.225576 (+ 312us) server_negotiation.cc:299] Negotiation successful
0620 11:34:10.225668 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 11:34:10.226032 13726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:10.216617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:35141 (local address 127.2.243.131:59227)
0620 11:34:10.216777 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:10.216820 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:10.217294 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:10.217503 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:10.217506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:10.218076 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:10.218265 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:10.218269 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.218982 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.218983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:10.219797 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:10.219799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.220130 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.220131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:10.220229 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:10.220833 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:10.220845 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:10.222656 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:34:10.224290 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:34:10.224297 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:34:10.224299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:34:10.224556 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:34:10.224809 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:34:10.224810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:34:10.224810 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:34:10.224936 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:34:10.225254 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:34:10.225261 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:34:10.225419 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 11:34:10.225537 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 11:34:10.225771 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260620 11:34:10.226555 13724 heartbeater.cc:347] Connected to a master server at 127.2.243.190:35141
I20260620 11:34:10.226682 13724 heartbeater.cc:464] Registering TS with master...
I20260620 11:34:10.226981 13724 heartbeater.cc:511] Master 127.2.243.190:35141 requested a full tablet report, sending...
I20260620 11:34:10.227423 13265 ts_manager.cc:194] Registered new tserver with Master: ec49a2c90ae04a588173de1706d52a6b (127.2.243.131:41901)
I20260620 11:34:10.227782 13265 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:59227
I20260620 11:34:10.233323 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:34:10.240831 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:10.234643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42864 (local address 127.2.243.190:35141)
0620 11:34:10.234735 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:10.234737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:10.234909 (+ 172us) server_negotiation.cc:407] Connection header received
0620 11:34:10.235000 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:10.235001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:10.235010 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:10.235061 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:10.235596 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.236086 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.236785 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.237020 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.237087 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:10.238242 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:34:10.238243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:34:10.238243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:34:10.238249 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:34:10.239759 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:34:10.240104 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:34:10.240104 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:34:10.240105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:34:10.240158 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:34:10.240335 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:34:10.240335 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:34:10.240335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:34:10.240478 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:34:10.240553 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:10.240672 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 11:34:10.240764 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 11:34:10.242651 13265 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:34:10.244387 13265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:34:10.251050 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:10.247123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37038 (local address 127.2.243.130:35085)
0620 11:34:10.247401 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:10.247406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:10.247415 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:34:10.247452 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:10.247454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:10.247469 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:10.247673 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:34:10.247743 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.248779 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.249871 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.250590 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.250674 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:10.250731 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:10.250919 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 11:34:10.250960 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":104,"threads_started":1}
I20260620 11:34:10.251083 13735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:10.246727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:35085 (local address 127.0.0.1:37038)
0620 11:34:10.246891 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:10.246932 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:10.246996 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:10.247566 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:10.247567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:10.247569 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:34:10.247605 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:10.247606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.248880 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.248881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:10.249788 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:10.249790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.250693 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.250695 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:10.250780 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:10.250796 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 11:34:10.250903 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 11:34:10.250981 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":2,"thread_start_us":86,"threads_started":1}
I20260620 11:34:10.251295 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:10.246853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37274 (local address 127.2.243.129:45805)
0620 11:34:10.247070 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:10.247075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:10.247084 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:34:10.247122 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:10.247125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:10.247140 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:10.247216 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:34:10.247468 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.248616 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.249984 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.250613 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.250674 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:10.250731 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:10.251169 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 11:34:10.251209 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":109,"threads_started":1}
I20260620 11:34:10.251083 13736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:10.246718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:45805 (local address 127.0.0.1:37274)
0620 11:34:10.246890 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:10.246931 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:10.246996 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:10.247311 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:10.247314 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:10.247316 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:34:10.247385 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:10.247387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.248751 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.248752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:10.249898 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:10.249901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.250695 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.250696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:10.250786 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:10.250796 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 11:34:10.250903 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 11:34:10.250982 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":41,"thread_start_us":51,"threads_started":1}
I20260620 11:34:10.251866 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:10.246961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:41901 (local address 127.0.0.1:53320)
0620 11:34:10.247207 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:10.247217 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:10.247268 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:10.247620 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:10.247621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:10.247623 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:34:10.247674 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:10.247676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.248567 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.248568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:10.249900 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:10.249902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.251647 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.251649 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:10.251720 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:10.251724 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:34:10.251764 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 11:34:10.251803 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
I20260620 11:34:10.251996 13524 tablet_service.cc:1467] Processing CreateTablet for tablet 220f93fdebe14329b87ce23fe06f2eaf (DEFAULT_TABLE table=test-table [id=89b5bd06de844bb8a6e983744b9e4444]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:34:10.252250 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:10.247235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53320 (local address 127.2.243.131:41901)
0620 11:34:10.247382 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:10.247386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:10.247396 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:34:10.247436 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:10.247438 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:10.247453 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:10.247758 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:34:10.247823 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.248461 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.250779 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.251544 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.252034 (+ 490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:10.252102 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:10.252127 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 11:34:10.252173 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
I20260620 11:34:10.252194 13390 tablet_service.cc:1467] Processing CreateTablet for tablet 220f93fdebe14329b87ce23fe06f2eaf (DEFAULT_TABLE table=test-table [id=89b5bd06de844bb8a6e983744b9e4444]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:34:10.252403 13524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220f93fdebe14329b87ce23fe06f2eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:34:10.252573 13390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220f93fdebe14329b87ce23fe06f2eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:34:10.252942 13658 tablet_service.cc:1467] Processing CreateTablet for tablet 220f93fdebe14329b87ce23fe06f2eaf (DEFAULT_TABLE table=test-table [id=89b5bd06de844bb8a6e983744b9e4444]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:34:10.253317 13658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 220f93fdebe14329b87ce23fe06f2eaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:34:10.255440 13741 tablet_bootstrap.cc:492] T 220f93fdebe14329b87ce23fe06f2eaf P 39fd7cfff83c49bd94059cf347ae2bb1: Bootstrap starting.
I20260620 11:34:10.256039 13742 tablet_bootstrap.cc:492] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455: Bootstrap starting.
I20260620 11:34:10.256222 13741 tablet_bootstrap.cc:654] T 220f93fdebe14329b87ce23fe06f2eaf P 39fd7cfff83c49bd94059cf347ae2bb1: Neither blocks nor log segments found. Creating new log.
I20260620 11:34:10.256731 13742 tablet_bootstrap.cc:654] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455: Neither blocks nor log segments found. Creating new log.
I20260620 11:34:10.256779 13743 tablet_bootstrap.cc:492] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b: Bootstrap starting.
I20260620 11:34:10.256949 13741 log.cc:824] T 220f93fdebe14329b87ce23fe06f2eaf P 39fd7cfff83c49bd94059cf347ae2bb1: Log is configured to *not* fsync() on all Append() calls
I20260620 11:34:10.257289 13742 log.cc:824] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455: Log is configured to *not* fsync() on all Append() calls
I20260620 11:34:10.257494 13743 tablet_bootstrap.cc:654] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b: Neither blocks nor log segments found. Creating new log.
I20260620 11:34:10.257833 13741 tablet_bootstrap.cc:492] T 220f93fdebe14329b87ce23fe06f2eaf P 39fd7cfff83c49bd94059cf347ae2bb1: No bootstrap required, opened a new log
I20260620 11:34:10.257889 13741 ts_tablet_manager.cc:1397] T 220f93fdebe14329b87ce23fe06f2eaf P 39fd7cfff83c49bd94059cf347ae2bb1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:34:10.258031 13743 log.cc:824] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b: Log is configured to *not* fsync() on all Append() calls
I20260620 11:34:10.258806 13743 tablet_bootstrap.cc:492] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b: No bootstrap required, opened a new log
I20260620 11:34:10.258926 13743 ts_tablet_manager.cc:1397] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:34:10.259465 13742 tablet_bootstrap.cc:492] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455: No bootstrap required, opened a new log
I20260620 11:34:10.259539 13742 ts_tablet_manager.cc:1397] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 11:34:10.259984 13741 raft_consensus.cc:348] T 220f93fdebe14329b87ce23fe06f2eaf P 39fd7cfff83c49bd94059cf347ae2bb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c6eb5e1100421d9f49a7fd9cb20455" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 35085 } } peers { permanent_uuid: "39fd7cfff83c49bd94059cf347ae2bb1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45805 } } peers { permanent_uuid: "ec49a2c90ae04a588173de1706d52a6b" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 41901 } }
I20260620 11:34:10.260160 13741 raft_consensus.cc:374] T 220f93fdebe14329b87ce23fe06f2eaf P 39fd7cfff83c49bd94059cf347ae2bb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:34:10.260192 13741 raft_consensus.cc:729] T 220f93fdebe14329b87ce23fe06f2eaf P 39fd7cfff83c49bd94059cf347ae2bb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39fd7cfff83c49bd94059cf347ae2bb1, State: Initialized, Role: FOLLOWER
I20260620 11:34:10.260337 13741 consensus_queue.cc:260] T 220f93fdebe14329b87ce23fe06f2eaf P 39fd7cfff83c49bd94059cf347ae2bb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c6eb5e1100421d9f49a7fd9cb20455" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 35085 } } peers { permanent_uuid: "39fd7cfff83c49bd94059cf347ae2bb1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45805 } } peers { permanent_uuid: "ec49a2c90ae04a588173de1706d52a6b" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 41901 } }
I20260620 11:34:10.260624 13741 ts_tablet_manager.cc:1428] T 220f93fdebe14329b87ce23fe06f2eaf P 39fd7cfff83c49bd94059cf347ae2bb1: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:34:10.260756 13456 heartbeater.cc:503] Master 127.2.243.190:35141 was elected leader, sending a full tablet report...
I20260620 11:34:10.260746 13743 raft_consensus.cc:348] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c6eb5e1100421d9f49a7fd9cb20455" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 35085 } } peers { permanent_uuid: "39fd7cfff83c49bd94059cf347ae2bb1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45805 } } peers { permanent_uuid: "ec49a2c90ae04a588173de1706d52a6b" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 41901 } }
I20260620 11:34:10.260898 13743 raft_consensus.cc:374] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:34:10.260928 13743 raft_consensus.cc:729] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec49a2c90ae04a588173de1706d52a6b, State: Initialized, Role: FOLLOWER
I20260620 11:34:10.261054 13743 consensus_queue.cc:260] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c6eb5e1100421d9f49a7fd9cb20455" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 35085 } } peers { permanent_uuid: "39fd7cfff83c49bd94059cf347ae2bb1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45805 } } peers { permanent_uuid: "ec49a2c90ae04a588173de1706d52a6b" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 41901 } }
I20260620 11:34:10.261354 13724 heartbeater.cc:503] Master 127.2.243.190:35141 was elected leader, sending a full tablet report...
I20260620 11:34:10.261348 13743 ts_tablet_manager.cc:1428] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:34:10.261699 13742 raft_consensus.cc:348] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c6eb5e1100421d9f49a7fd9cb20455" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 35085 } } peers { permanent_uuid: "39fd7cfff83c49bd94059cf347ae2bb1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45805 } } peers { permanent_uuid: "ec49a2c90ae04a588173de1706d52a6b" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 41901 } }
I20260620 11:34:10.261868 13742 raft_consensus.cc:374] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:34:10.261919 13742 raft_consensus.cc:729] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29c6eb5e1100421d9f49a7fd9cb20455, State: Initialized, Role: FOLLOWER
I20260620 11:34:10.262038 13742 consensus_queue.cc:260] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c6eb5e1100421d9f49a7fd9cb20455" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 35085 } } peers { permanent_uuid: "39fd7cfff83c49bd94059cf347ae2bb1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45805 } } peers { permanent_uuid: "ec49a2c90ae04a588173de1706d52a6b" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 41901 } }
I20260620 11:34:10.262377 13742 ts_tablet_manager.cc:1428] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:34:10.262447 13590 heartbeater.cc:503] Master 127.2.243.190:35141 was elected leader, sending a full tablet report...
W20260620 11:34:10.300599 13591 tablet.cc:2369] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 11:34:10.385675 13457 tablet.cc:2369] T 220f93fdebe14329b87ce23fe06f2eaf P 39fd7cfff83c49bd94059cf347ae2bb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:34:10.395033 13749 raft_consensus.cc:482] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:34:10.395160 13749 raft_consensus.cc:504] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c6eb5e1100421d9f49a7fd9cb20455" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 35085 } } peers { permanent_uuid: "39fd7cfff83c49bd94059cf347ae2bb1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45805 } } peers { permanent_uuid: "ec49a2c90ae04a588173de1706d52a6b" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 41901 } }
I20260620 11:34:10.395627 13749 leader_election.cc:288] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 39fd7cfff83c49bd94059cf347ae2bb1 (127.2.243.129:45805), ec49a2c90ae04a588173de1706d52a6b (127.2.243.131:41901)
I20260620 11:34:10.398797 13748 raft_consensus.cc:482] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:34:10.398936 13748 raft_consensus.cc:504] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c6eb5e1100421d9f49a7fd9cb20455" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 35085 } } peers { permanent_uuid: "39fd7cfff83c49bd94059cf347ae2bb1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45805 } } peers { permanent_uuid: "ec49a2c90ae04a588173de1706d52a6b" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 41901 } }
I20260620 11:34:10.399329 13748 leader_election.cc:288] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 29c6eb5e1100421d9f49a7fd9cb20455 (127.2.243.130:35085), 39fd7cfff83c49bd94059cf347ae2bb1 (127.2.243.129:45805)
I20260620 11:34:10.399724 13592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:10.395898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:45805 (local address 127.2.243.130:45913)
0620 11:34:10.395992 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:10.396005 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:10.396059 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:10.396220 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:10.396222 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:10.396224 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:34:10.396305 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:10.396308 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.397292 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.397293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:10.398568 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:10.398570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.399428 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.399429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:10.399522 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:10.399526 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:34:10.399578 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 11:34:10.399629 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 11:34:10.399830 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:10.395983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:56013 (local address 127.2.243.131:41901)
0620 11:34:10.396035 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:10.396037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:10.396381 (+ 344us) server_negotiation.cc:407] Connection header received
0620 11:34:10.396515 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:10.396522 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:10.396533 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:10.396592 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:34:10.396755 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.397382 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.398687 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.399355 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.399433 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:10.399443 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:10.399663 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 11:34:10.399752 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 11:34:10.400025 13737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:10.395969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:45913 (local address 127.2.243.129:45805)
0620 11:34:10.396020 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:10.396023 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:10.396041 (+ 18us) server_negotiation.cc:407] Connection header received
0620 11:34:10.396139 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:10.396141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:10.396152 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:10.396212 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:34:10.396394 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.397159 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.398706 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.399258 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.399341 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:10.399352 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:10.399893 (+ 541us) server_negotiation.cc:299] Negotiation successful
0620 11:34:10.399949 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:34:10.400364 13750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:10.395898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:41901 (local address 127.2.243.130:56013)
0620 11:34:10.396293 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:10.396302 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:10.396388 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:10.396609 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:10.396616 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:10.396618 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:34:10.396682 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:10.396683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.397468 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.397469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:10.398568 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:10.398570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.399492 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.399493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:10.399573 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:10.399578 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:34:10.400236 (+ 658us) client_negotiation.cc:240] Negotiation successful
0620 11:34:10.400287 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":156,"thread_start_us":152,"threads_started":1}
I20260620 11:34:10.400386 13410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220f93fdebe14329b87ce23fe06f2eaf" candidate_uuid: "29c6eb5e1100421d9f49a7fd9cb20455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39fd7cfff83c49bd94059cf347ae2bb1" is_pre_election: true
I20260620 11:34:10.400586 13410 raft_consensus.cc:2454] T 220f93fdebe14329b87ce23fe06f2eaf P 39fd7cfff83c49bd94059cf347ae2bb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29c6eb5e1100421d9f49a7fd9cb20455 in term 0.
I20260620 11:34:10.400681 13678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220f93fdebe14329b87ce23fe06f2eaf" candidate_uuid: "29c6eb5e1100421d9f49a7fd9cb20455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec49a2c90ae04a588173de1706d52a6b" is_pre_election: true
I20260620 11:34:10.400777 13678 raft_consensus.cc:2454] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29c6eb5e1100421d9f49a7fd9cb20455 in term 0.
I20260620 11:34:10.400846 13477 leader_election.cc:302] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29c6eb5e1100421d9f49a7fd9cb20455, 39fd7cfff83c49bd94059cf347ae2bb1; no voters:
I20260620 11:34:10.401216 13749 raft_consensus.cc:2781] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:34:10.401304 13749 raft_consensus.cc:482] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:34:10.401335 13749 raft_consensus.cc:3037] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:34:10.402253 13749 raft_consensus.cc:504] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c6eb5e1100421d9f49a7fd9cb20455" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 35085 } } peers { permanent_uuid: "39fd7cfff83c49bd94059cf347ae2bb1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45805 } } peers { permanent_uuid: "ec49a2c90ae04a588173de1706d52a6b" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 41901 } }
I20260620 11:34:10.402523 13410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220f93fdebe14329b87ce23fe06f2eaf" candidate_uuid: "29c6eb5e1100421d9f49a7fd9cb20455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39fd7cfff83c49bd94059cf347ae2bb1"
I20260620 11:34:10.402577 13678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220f93fdebe14329b87ce23fe06f2eaf" candidate_uuid: "29c6eb5e1100421d9f49a7fd9cb20455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec49a2c90ae04a588173de1706d52a6b"
I20260620 11:34:10.402606 13410 raft_consensus.cc:3037] T 220f93fdebe14329b87ce23fe06f2eaf P 39fd7cfff83c49bd94059cf347ae2bb1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:34:10.402635 13678 raft_consensus.cc:3037] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:34:10.403155 13726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:10.399483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:35085 (local address 127.2.243.131:50539)
0620 11:34:10.399549 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:10.399561 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:10.399596 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:10.400109 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:10.400110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:10.400112 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:34:10.400162 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:10.400164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.401048 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.401049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:10.402140 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:10.402143 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.402840 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.402841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:10.402969 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:10.402972 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:34:10.403023 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 11:34:10.403073 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":10}
I20260620 11:34:10.403244 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:10.399739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:38971 (local address 127.2.243.129:45805)
0620 11:34:10.399984 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:10.399986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:10.399997 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:34:10.400017 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:10.400017 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:10.400026 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:10.400085 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:34:10.400313 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.401109 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.402108 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.402726 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.402800 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:10.402807 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:10.403111 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 11:34:10.403151 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":55,"threads_started":1}
I20260620 11:34:10.403398 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:10.399945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:50539 (local address 127.2.243.130:35085)
0620 11:34:10.399979 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:10.399981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:10.399993 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:34:10.400012 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:10.400013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:10.400021 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:10.400072 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:34:10.400248 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.400965 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.402229 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.402740 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.402799 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:10.402806 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:10.403312 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 11:34:10.403360 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":18}
I20260620 11:34:10.403510 13410 raft_consensus.cc:2454] T 220f93fdebe14329b87ce23fe06f2eaf P 39fd7cfff83c49bd94059cf347ae2bb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29c6eb5e1100421d9f49a7fd9cb20455 in term 1.
I20260620 11:34:10.403510 13678 raft_consensus.cc:2454] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29c6eb5e1100421d9f49a7fd9cb20455 in term 1.
I20260620 11:34:10.403616 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:10.399511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:45805 (local address 127.2.243.131:38971)
0620 11:34:10.399751 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:10.399760 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:10.399809 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:10.400123 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:10.400124 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:10.400126 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:34:10.400190 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:10.400192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.401195 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.401196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:10.402008 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:10.402010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.402887 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.402888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:10.402969 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:10.402972 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:34:10.403023 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 11:34:10.403072 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":14,"thread_start_us":98,"threads_started":1}
I20260620 11:34:10.403692 13477 leader_election.cc:302] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29c6eb5e1100421d9f49a7fd9cb20455, 39fd7cfff83c49bd94059cf347ae2bb1; no voters:
I20260620 11:34:10.403679 13544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220f93fdebe14329b87ce23fe06f2eaf" candidate_uuid: "ec49a2c90ae04a588173de1706d52a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "29c6eb5e1100421d9f49a7fd9cb20455" is_pre_election: true
I20260620 11:34:10.403812 13544 raft_consensus.cc:2379] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec49a2c90ae04a588173de1706d52a6b in current term 1: Already voted for candidate 29c6eb5e1100421d9f49a7fd9cb20455 in this term.
I20260620 11:34:10.404044 13749 leader_election.cc:288] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 [CANDIDATE]: Term 1 election: Requested vote from peers 39fd7cfff83c49bd94059cf347ae2bb1 (127.2.243.129:45805), ec49a2c90ae04a588173de1706d52a6b (127.2.243.131:41901)
I20260620 11:34:10.404105 13749 raft_consensus.cc:2781] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:34:10.404143 13749 raft_consensus.cc:686] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 [term 1 LEADER]: Becoming Leader. State: Replica: 29c6eb5e1100421d9f49a7fd9cb20455, State: Running, Role: LEADER
I20260620 11:34:10.404219 13749 consensus_queue.cc:237] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c6eb5e1100421d9f49a7fd9cb20455" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 35085 } } peers { permanent_uuid: "39fd7cfff83c49bd94059cf347ae2bb1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45805 } } peers { permanent_uuid: "ec49a2c90ae04a588173de1706d52a6b" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 41901 } }
I20260620 11:34:10.404307 13410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "220f93fdebe14329b87ce23fe06f2eaf" candidate_uuid: "ec49a2c90ae04a588173de1706d52a6b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39fd7cfff83c49bd94059cf347ae2bb1" is_pre_election: true
I20260620 11:34:10.404404 13410 raft_consensus.cc:2379] T 220f93fdebe14329b87ce23fe06f2eaf P 39fd7cfff83c49bd94059cf347ae2bb1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ec49a2c90ae04a588173de1706d52a6b in current term 1: Already voted for candidate 29c6eb5e1100421d9f49a7fd9cb20455 in this term.
I20260620 11:34:10.404608 13611 leader_election.cc:302] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ec49a2c90ae04a588173de1706d52a6b; no voters: 29c6eb5e1100421d9f49a7fd9cb20455, 39fd7cfff83c49bd94059cf347ae2bb1
I20260620 11:34:10.404753 13748 raft_consensus.cc:2726] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 11:34:10.405088 13265 catalog_manager.cc:5495] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29c6eb5e1100421d9f49a7fd9cb20455 (127.2.243.130). New cstate: current_term: 1 leader_uuid: "29c6eb5e1100421d9f49a7fd9cb20455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "29c6eb5e1100421d9f49a7fd9cb20455" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 35085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39fd7cfff83c49bd94059cf347ae2bb1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec49a2c90ae04a588173de1706d52a6b" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 41901 } health_report { overall_health: UNKNOWN } } }
I20260620 11:34:10.437367 13739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:10.435190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.2.243.130:35085)
0620 11:34:10.435250 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:10.435252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:10.435268 (+ 16us) server_negotiation.cc:407] Connection header received
0620 11:34:10.435301 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:10.435301 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:10.435318 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:10.435379 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:34:10.435495 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.435970 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.436659 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:10.436833 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:10.436883 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:10.437012 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:34:10.437115 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:34:10.437184 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:10.437236 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 11:34:10.437252 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 11:34:10.439206 13410 raft_consensus.cc:1261] T 220f93fdebe14329b87ce23fe06f2eaf P 39fd7cfff83c49bd94059cf347ae2bb1 [term 1 FOLLOWER]: Refusing update from remote peer 29c6eb5e1100421d9f49a7fd9cb20455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:34:10.439232 13678 raft_consensus.cc:1261] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b [term 1 FOLLOWER]: Refusing update from remote peer 29c6eb5e1100421d9f49a7fd9cb20455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:34:10.439561 13749 consensus_queue.cc:1035] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39fd7cfff83c49bd94059cf347ae2bb1" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:34:10.439697 13753 consensus_queue.cc:1035] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec49a2c90ae04a588173de1706d52a6b" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:34:10.442781 13758 mvcc.cc:205] Tried to move back new op lower bound from 7298888705795899392 to 7298888705656229888. Current Snapshot: MvccSnapshot[applied={T|T < 7298888705795899392}]
I20260620 11:34:10.445040 13264 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:34:10.445151 13264 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:34:10.445938 13264 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8bd8f12e190245baa0a49851bf79d29d: Sending DeleteTablet for 3 replicas of tablet 220f93fdebe14329b87ce23fe06f2eaf
I20260620 11:34:10.446256 13390 tablet_service.cc:1514] Processing DeleteTablet for tablet 220f93fdebe14329b87ce23fe06f2eaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:34:10 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:37274
I20260620 11:34:10.446293 13658 tablet_service.cc:1514] Processing DeleteTablet for tablet 220f93fdebe14329b87ce23fe06f2eaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:34:10 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:53320
I20260620 11:34:10.446307 13524 tablet_service.cc:1514] Processing DeleteTablet for tablet 220f93fdebe14329b87ce23fe06f2eaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:34:10 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:37038
I20260620 11:34:10.446564 13766 tablet_replica.cc:330] T 220f93fdebe14329b87ce23fe06f2eaf P 39fd7cfff83c49bd94059cf347ae2bb1: stopping tablet replica
I20260620 11:34:10.446668 13766 raft_consensus.cc:2229] T 220f93fdebe14329b87ce23fe06f2eaf P 39fd7cfff83c49bd94059cf347ae2bb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:34:10.446744 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 13327
I20260620 11:34:10.446791 13766 raft_consensus.cc:2258] T 220f93fdebe14329b87ce23fe06f2eaf P 39fd7cfff83c49bd94059cf347ae2bb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:34:10.451292 13768 tablet_replica.cc:330] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b: stopping tablet replica
I20260620 11:34:10.451490 13768 raft_consensus.cc:2229] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:34:10.451494 13767 tablet_replica.cc:330] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455: stopping tablet replica
I20260620 11:34:10.451597 13767 raft_consensus.cc:2229] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:34:10.451719 13768 raft_consensus.cc:2258] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:34:10.451743 13767 raft_consensus.cc:2258] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:34:10.452216 13767 ts_tablet_manager.cc:1905] T 220f93fdebe14329b87ce23fe06f2eaf P 29c6eb5e1100421d9f49a7fd9cb20455: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:34:10.452411 13768 ts_tablet_manager.cc:1905] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 11:34:10.452875 13249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.243.129:45805 (error 108)
I20260620 11:34:10.453531 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 13461
I20260620 11:34:10.455241 13738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:10.454969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:45805 (local address 127.0.0.1:37276)
0620 11:34:10.455177 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:10.455210 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.129:45805: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
W20260620 11:34:10.455399 13249 catalog_manager.cc:4584] TS 39fd7cfff83c49bd94059cf347ae2bb1 (127.2.243.129:45805): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 220f93fdebe14329b87ce23fe06f2eaf: Network error: Client connection negotiation failed: client connection to 127.2.243.129:45805: connect: Connection refused (error 111)
I20260620 11:34:10.456122 13768 ts_tablet_manager.cc:1918] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:34:10.456228 13768 log.cc:1196] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/220f93fdebe14329b87ce23fe06f2eaf
I20260620 11:34:10.456753 13768 ts_tablet_manager.cc:1939] T 220f93fdebe14329b87ce23fe06f2eaf P ec49a2c90ae04a588173de1706d52a6b: Deleting consensus metadata
I20260620 11:34:10.457193 13251 catalog_manager.cc:4841] TS ec49a2c90ae04a588173de1706d52a6b (127.2.243.131:41901): tablet 220f93fdebe14329b87ce23fe06f2eaf (table test-table [id=89b5bd06de844bb8a6e983744b9e4444]) successfully deleted
W20260620 11:34:10.458853 13738 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:34:10.458728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:35085 (local address 127.0.0.1:37062)
0620 11:34:10.458784 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:10.458812 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.130:35085: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 11:34:10.459081 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 13595
I20260620 11:34:10.464226 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 13233
2026-06-20T11:34:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2121 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:34:10 dist-test-slave-ctbw krb5kdc[13772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:34:10 dist-test-slave-ctbw krb5kdc[13772](info): set up 2 sockets
Jun 20 11:34:10 dist-test-slave-ctbw krb5kdc[13772](info): commencing operation
krb5kdc: starting...
W20260620 11:34:11.512084 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:34:11 dist-test-slave-ctbw krb5kdc[13772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:34:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:34:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/rangeradmin_127.2.243.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/rangeradmin_127.2.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/rangerlookup_127.2.243.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/rangerlookup_127.2.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/HTTP_127.2.243.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/HTTP_127.2.243.148@KRBTEST.COM.keytab.
I20260620 11:34:11.681530 3022 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskDepOyZ/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/postgres -l logfile start
2026-06-20 11:34:14.137 UTC [13802] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 11:34:14.137 UTC [13802] LOG: listening on IPv4 address "127.2.243.148", port 59853
2026-06-20 11:34:14.141 UTC [13802] LOG: listening on Unix socket "/tmp/.s.PGSQL.59853"
2026-06-20 11:34:14.169 UTC [13805] LOG: database system was shut down at 2026-06-20 11:34:12 UTC
2026-06-20 11:34:14.172 UTC [13802] LOG: database system is ready to accept connections
I20260620 11:34:15.132407 3022 mini_postgres.cc:96] Postgres bound to 59853
2026-06-20 11:34:15.139 UTC [13813] FATAL: database "slave" does not exist
127.2.243.148:59853 - accepting connections
I20260620 11:34:15.139621 3022 mini_ranger.cc:161] Starting Ranger...
I20260620 11:34:15.156057 3022 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 11:34:15.615326 3022 mini_ranger.cc:87] Created ranger Postgres database
I20260620 11:34:15.615415 3022 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ranger-admin
2026-06-20 11:34:15,910 [I] DB FLAVOR :POSTGRES
2026-06-20 11:34:15,911 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 11:34:15,911 [I] Checking connection..
2026-06-20 11:34:15,911 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 11:34:16,186 [I] Checking connection passed.
2026-06-20 11:34:16,186 [I] --------- Verifying version history table ---------
2026-06-20 11:34:16,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 11:34:16,429 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 11:34:16,429 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 11:34:16,680 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 11:34:16,681 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 11:34:16,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 11:34:16.918 UTC [13877] WARNING: there is no transaction in progress
2026-06-20 11:34:16,924 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 11:34:16,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 11:34:17,175 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 11:34:17,176 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 11:34:17,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 11:34:17,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 11:34:17,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ctbw.c.gcp-upstream.internal','N') ;"
2026-06-20 11:34:17,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 11:34:18,161 [I] Table x_portal_user does not exist in database ranger
2026-06-20 11:34:18,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 11:34:18,396 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 11:34:18,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 11:34:18,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 11:34:18,907 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 11:34:18,908 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 11:34:19.932 UTC [13985] WARNING: there is no transaction in progress
2026-06-20 11:34:19.948 UTC [13985] WARNING: there is no transaction in progress
2026-06-20 11:34:19.965 UTC [13985] WARNING: there is no transaction in progress
2026-06-20 11:34:19.981 UTC [13985] WARNING: there is no transaction in progress
2026-06-20 11:34:19.998 UTC [13985] WARNING: there is no transaction in progress
2026-06-20 11:34:20.018 UTC [13985] WARNING: there is no transaction in progress
2026-06-20 11:34:20.073 UTC [13985] WARNING: there is no transaction in progress
2026-06-20 11:34:20.080 UTC [13985] WARNING: there is no transaction in progress
2026-06-20 11:34:20.096 UTC [13985] WARNING: there is no transaction in progress
2026-06-20 11:34:20.107 UTC [13985] WARNING: there is no transaction in progress
2026-06-20 11:34:20.117 UTC [13985] WARNING: there is no transaction in progress
2026-06-20 11:34:20.131 UTC [13985] WARNING: there is no transaction in progress
2026-06-20 11:34:20.141 UTC [13985] WARNING: there is no transaction in progress
2026-06-20 11:34:20.148 UTC [13985] WARNING: there is no transaction in progress
2026-06-20 11:34:20.155 UTC [13985] WARNING: there is no transaction in progress
2026-06-20 11:34:20.161 UTC [13985] WARNING: there is no transaction in progress
2026-06-20 11:34:20,809 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 11:34:20,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 11:34:21,041 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 11:34:21,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 11:34:21,289 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 11:34:21,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 11:34:21,525 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 11:34:21,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 11:34:21,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 11:34:22,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ctbw.c.gcp-upstream.internal';"
2026-06-20 11:34:22,279 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 11:34:22,280 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 11:34:22,515 [I] DB_PATCHES have already been applied
I20260620 11:34:22.521063 3022 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ranger-admin:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskDepOyZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 11:34:22.521130 3022 mini_ranger.cc:193] Using host: 127.2.243.148
I20260620 11:34:22.524326 3022 mini_ranger.cc:227] Ranger admin URL: http://127.2.243.148:41869
Jun 20, 2026 11:34:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:34:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:34:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 11:34:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 11:34:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 11:34:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 11:34:23 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 11:34:23.659554 3022 mini_ranger.cc:160] Time spent starting Ranger: real 8.520s user 0.000s sys 0.009s
Jun 20, 2026 11:34:43 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 11:34:43 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 11:34:43 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 11:34:43 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 11:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 11:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 11:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 11:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 11:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 11:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 11:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 11:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 11:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 11:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 11:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 11:34:43 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 11:34:43 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 11:34:44 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 11:34:44.932682 3022 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/rangerkms_127.2.243.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/rangerkms_127.2.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.243.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.243.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.243.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/HTTP_127.2.243.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.243.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/HTTP_127.2.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 11:34:45 dist-test-slave-ctbw krb5kdc[13772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955285, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 11:34:45.028604 3022 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 11:34:45.040274 3022 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 11:34:45.864701 3022 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 11:34:45.864789 3022 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ranger-kms
I20260620 11:34:45.864809 3022 mini_ranger_kms.cc:227] Using postgres at 127.2.243.148:59853
2026-06-20 11:34:46,056 [I] DB FLAVOR :POSTGRES
2026-06-20 11:34:46,057 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 11:34:46,057 [I] Checking connection
2026-06-20 11:34:46,057 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 11:34:46,321 [I] connection success
2026-06-20 11:34:46,321 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 11:34:46,321 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 11:34:46,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 11:34:46,605 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 11:34:46,605 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 11:34:46,605 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 11:34:46,605 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:59853/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 11:34:46,942 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 11:34:47.407291 3022 mini_ranger_kms.cc:310] Created kms service
I20260620 11:34:47.629928 3022 mini_ranger_kms.cc:326] Created kudu user
I20260620 11:34:47.676621 3022 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 11:34:47 dist-test-slave-ctbw krb5kdc[13772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955285, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.243.148@KRBTEST.COM
I20260620 11:34:48.111426 3022 mini_ranger_kms.cc:384] Added ranger policy
I20260620 11:34:48.111562 3022 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ranger-kms:/tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 11:34:48.111586 3022 mini_ranger_kms.cc:240] Using host: 127.2.243.148
I20260620 11:34:48.114956 3022 mini_ranger_kms.cc:276] Ranger KMS PID: 14165
I20260620 11:34:48.115093 3022 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.243.148:41733
Jun 20, 2026 11:34:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:34:49 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:34:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 11:34:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 11:34:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 11:34:49 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 11:34:50.439468 3022 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.411s user 0.013s sys 0.161s
I20260620 11:34:50.439589 3022 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 11:34:50.439673 3022 mini_ranger_kms.cc:401] 127.2.243.148:41733/kms/v1/keys
Jun 20 11:34:50 dist-test-slave-ctbw krb5kdc[13772](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781955290, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.243.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 11:34:50 dist-test-slave-ctbw krb5kdc[13772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955290, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.243.148@KRBTEST.COM for HTTP/127.2.243.148@KRBTEST.COM
Jun 20 11:34:51 dist-test-slave-ctbw krb5kdc[13772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955290, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.243.148@KRBTEST.COM for HTTP/127.2.243.148@KRBTEST.COM
Jun 20 11:34:53 dist-test-slave-ctbw krb5kdc[13772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:34:53.657404 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:38067
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:45709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:38067
--ranger_config_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.243.148:41733/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:34:53.745237 14216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:34:53.745464 14216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:34:53.745488 14216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:34:53.746713 14216 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:34:53.746766 14216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:34:53.746780 14216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:34:53.746793 14216 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:34:53.746806 14216 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:34:53.748441 14216 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:45709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.243.148:41733/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:38067
--ranger_config_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:38067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:34:53.748695 14216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:34:53.748940 14216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:34:53.754321 14224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:53.754384 14221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:53.754506 14222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:34:53.754398 14216 server_base.cc:1034] running on GCE node
I20260620 11:34:53.754909 14216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:34:53.755292 14216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:34:53.756469 14216 hybrid_clock.cc:648] HybridClock initialized: now 1781955293756455 us; error 29 us; skew 500 ppm
Jun 20 11:34:53 dist-test-slave-ctbw krb5kdc[13772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955293, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:34:53.759099 14216 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:34:53.759621 14216 webserver.cc:460] Webserver started at http://127.2.243.190:39547/ using document root <none> and password file <none>
I20260620 11:34:53.759852 14216 fs_manager.cc:359] Metadata directory not provided
I20260620 11:34:53.759891 14216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:34:53.760035 14216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 11:34:53 dist-test-slave-ctbw krb5kdc[13772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955293, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for HTTP/127.2.243.148@KRBTEST.COM
I20260620 11:34:53.822690 14216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "a045875e5d904a26b3d977b164dc5abd"
format_stamp: "Formatted at 2026-06-20 11:34:53 on dist-test-slave-ctbw"
server_key: "446486d6eece395fc8f70e392c08e06d"
server_key_iv: "fb49f2513ca0692dd35e664a34be1cfa"
server_key_version: "kuduclusterkey@0"
I20260620 11:34:53.823160 14216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "a045875e5d904a26b3d977b164dc5abd"
format_stamp: "Formatted at 2026-06-20 11:34:53 on dist-test-slave-ctbw"
server_key: "446486d6eece395fc8f70e392c08e06d"
server_key_iv: "fb49f2513ca0692dd35e664a34be1cfa"
server_key_version: "kuduclusterkey@0"
I20260620 11:34:53.824854 14216 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:34:53.837178 14231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:53.837447 14216 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 11:34:53.837558 14216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "a045875e5d904a26b3d977b164dc5abd"
format_stamp: "Formatted at 2026-06-20 11:34:53 on dist-test-slave-ctbw"
server_key: "446486d6eece395fc8f70e392c08e06d"
server_key_iv: "fb49f2513ca0692dd35e664a34be1cfa"
server_key_version: "kuduclusterkey@0"
I20260620 11:34:53.837687 14216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:34:53.859683 14216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:34:53.861749 14216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:34:53.861908 14216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:34:53.867247 14216 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:38067
I20260620 11:34:53.867264 14283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:38067 every 8 connection(s)
I20260620 11:34:53.867668 14216 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:34:53.868461 14284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:34:53.871593 14284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd: Bootstrap starting.
I20260620 11:34:53.872331 14284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd: Neither blocks nor log segments found. Creating new log.
I20260620 11:34:53.872646 14284 log.cc:824] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd: Log is configured to *not* fsync() on all Append() calls
I20260620 11:34:53.873488 14284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd: No bootstrap required, opened a new log
I20260620 11:34:53.874310 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 14216
I20260620 11:34:53.874403 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:34:53.874647 3022 external_mini_cluster.cc:1428] Setting key 6e4eacfcc4e41375e2dd24130622ca47
I20260620 11:34:53.876816 14284 raft_consensus.cc:348] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a045875e5d904a26b3d977b164dc5abd" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 38067 } }
I20260620 11:34:53.877010 14284 raft_consensus.cc:374] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:34:53.877044 14284 raft_consensus.cc:729] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a045875e5d904a26b3d977b164dc5abd, State: Initialized, Role: FOLLOWER
I20260620 11:34:53.877170 14284 consensus_queue.cc:260] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a045875e5d904a26b3d977b164dc5abd" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 38067 } }
I20260620 11:34:53.877257 14284 raft_consensus.cc:388] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:34:53.877285 14284 raft_consensus.cc:482] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:34:53.877323 14284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:34:53.878000 14284 raft_consensus.cc:504] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a045875e5d904a26b3d977b164dc5abd" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 38067 } }
I20260620 11:34:53.878111 14284 leader_election.cc:302] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a045875e5d904a26b3d977b164dc5abd; no voters:
I20260620 11:34:53.878262 14284 leader_election.cc:288] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:34:53.878324 14289 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:34:53.878505 14284 sys_catalog.cc:564] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:34:53.878614 14289 raft_consensus.cc:686] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd [term 1 LEADER]: Becoming Leader. State: Replica: a045875e5d904a26b3d977b164dc5abd, State: Running, Role: LEADER
I20260620 11:34:53.878785 14289 consensus_queue.cc:237] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a045875e5d904a26b3d977b164dc5abd" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 38067 } }
I20260620 11:34:53.879215 14291 sys_catalog.cc:455] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a045875e5d904a26b3d977b164dc5abd. Latest consensus state: current_term: 1 leader_uuid: "a045875e5d904a26b3d977b164dc5abd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a045875e5d904a26b3d977b164dc5abd" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 38067 } } }
I20260620 11:34:53.879206 14290 sys_catalog.cc:455] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a045875e5d904a26b3d977b164dc5abd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a045875e5d904a26b3d977b164dc5abd" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 38067 } } }
I20260620 11:34:53.879304 14291 sys_catalog.cc:458] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd [sys.catalog]: This master's current role is: LEADER
I20260620 11:34:53.879307 14290 sys_catalog.cc:458] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd [sys.catalog]: This master's current role is: LEADER
I20260620 11:34:53.879701 14299 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:34:53.879911 14284 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 11:34:53.880499 14299 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:34:53.886011 14299 catalog_manager.cc:1347] Generated new cluster ID: 4c7f5d3144c94795b4fb8e9fcb35e03e
I20260620 11:34:53.886161 14299 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:34:53 dist-test-slave-ctbw krb5kdc[13772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955293, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:34:53.897928 14299 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:34:53.898844 14299 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:34:53.900444 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:53.875189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.2.243.190:38067)
0620 11:34:53.875351 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:53.875364 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:53.880154 (+ 4790us) server_negotiation.cc:407] Connection header received
0620 11:34:53.880328 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:53.880336 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:53.880415 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:53.880609 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:53.881331 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:53.885466 (+ 4135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:53.886600 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:53.887662 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:53.887791 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:53.893668 (+ 5877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:34:53.893675 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:34:53.893676 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:34:53.893693 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:34:53.897692 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:34:53.898141 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:34:53.898145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:34:53.898145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:34:53.898290 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:34:53.898647 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:34:53.898648 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:34:53.898648 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:34:53.899048 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:34:53.899216 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:53.899740 (+ 524us) server_negotiation.cc:299] Negotiation successful
0620 11:34:53.900144 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":82,"threads_started":1}
I20260620 11:34:53.912101 14299 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd: Generated new TSK 0
I20260620 11:34:53.912370 14299 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
I20260620 11:34:54.695654 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.243.148:41733/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:38067
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:45709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:34:54.805063 14330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:34:54.805354 14330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:34:54.805392 14330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:34:54.807231 14330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:34:54.807301 14330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:34:54.807345 14330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:34:54.809868 14330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:45709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.243.148:41733/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:38067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:34:54.810165 14330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:34:54.810487 14330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:34:54.817011 14336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:54.817055 14338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:34:54.817236 14330 server_base.cc:1034] running on GCE node
W20260620 11:34:54.817384 14335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:34:54.817660 14330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:34:54.817950 14330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:34:54.819119 14330 hybrid_clock.cc:648] HybridClock initialized: now 1781955294819088 us; error 47 us; skew 500 ppm
Jun 20 11:34:54 dist-test-slave-ctbw krb5kdc[13772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955294, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:34:54.825829 14330 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:34:54.826346 14330 webserver.cc:460] Webserver started at http://127.2.243.129:41627/ using document root <none> and password file <none>
I20260620 11:34:54.826606 14330 fs_manager.cc:359] Metadata directory not provided
I20260620 11:34:54.826687 14330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:34:54.826848 14330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 11:34:54 dist-test-slave-ctbw krb5kdc[13772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955294, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for HTTP/127.2.243.148@KRBTEST.COM
I20260620 11:34:54.839701 14330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "65dfcb6df4ba4b12a639b41c809622c9"
format_stamp: "Formatted at 2026-06-20 11:34:54 on dist-test-slave-ctbw"
server_key: "365818c121784619de164c3a432fe6f4"
server_key_iv: "8c396da73bede4251e1e7ff3344410aa"
server_key_version: "kuduclusterkey@0"
I20260620 11:34:54.840242 14330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "65dfcb6df4ba4b12a639b41c809622c9"
format_stamp: "Formatted at 2026-06-20 11:34:54 on dist-test-slave-ctbw"
server_key: "365818c121784619de164c3a432fe6f4"
server_key_iv: "8c396da73bede4251e1e7ff3344410aa"
server_key_version: "kuduclusterkey@0"
I20260620 11:34:54.842195 14330 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 11:34:54.863035 14346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:54.863309 14330 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260620 11:34:54.863415 14330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "65dfcb6df4ba4b12a639b41c809622c9"
format_stamp: "Formatted at 2026-06-20 11:34:54 on dist-test-slave-ctbw"
server_key: "365818c121784619de164c3a432fe6f4"
server_key_iv: "8c396da73bede4251e1e7ff3344410aa"
server_key_version: "kuduclusterkey@0"
I20260620 11:34:54.863562 14330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:34:54.881156 14330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:34:54.883345 14330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:34:54.883507 14330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:34:54.883766 14330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:34:54.884151 14330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:34:54.884188 14330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:54.884218 14330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:34:54.884254 14330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:54.891553 14330 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:46749
I20260620 11:34:54.892004 14330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:34:54.895221 14460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:46749 every 8 connection(s)
Jun 20 11:34:54 dist-test-slave-ctbw krb5kdc[13772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955294, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:34:54.901738 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 14330
I20260620 11:34:54.901839 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:34:54.902078 3022 external_mini_cluster.cc:1428] Setting key 1c7232eb0b526c33f43c66106905ccde
I20260620 11:34:54.902222 14464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:54.893102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:39615 (local address 127.2.243.190:38067)
0620 11:34:54.893968 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:54.893970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:54.893977 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:34:54.893998 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:54.893998 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:54.894008 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:54.894098 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:54.894666 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:54.895434 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:54.896459 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:54.896677 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:54.896757 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:54.898841 (+ 2084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:34:54.898843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:34:54.898844 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:34:54.898853 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:34:54.900642 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:34:54.901035 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:34:54.901037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:34:54.901037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:34:54.901110 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:34:54.901394 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:34:54.901396 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:34:54.901396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:34:54.901545 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:34:54.901628 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:54.902012 (+ 384us) server_negotiation.cc:299] Negotiation successful
0620 11:34:54.902109 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":839,"thread_start_us":71,"threads_started":1}
I20260620 11:34:54.902370 14463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:54.892781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:38067 (local address 127.2.243.129:39615)
0620 11:34:54.893280 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:54.893334 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:54.893822 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:54.894176 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:54.894181 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:54.894375 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:54.894562 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:54.894566 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:54.895536 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:54.895537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:54.896347 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:54.896349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:54.896763 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:54.896764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:54.896898 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:54.897462 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:54.897471 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:54.898733 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:34:54.900726 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:34:54.900732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:34:54.900734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:34:54.900953 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:34:54.901219 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:34:54.901220 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:34:54.901220 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:34:54.901319 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:34:54.901629 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:34:54.901635 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:34:54.901847 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 11:34:54.901979 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 11:34:54.902136 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":78,"threads_started":1}
I20260620 11:34:54.903160 14461 heartbeater.cc:347] Connected to a master server at 127.2.243.190:38067
I20260620 11:34:54.903258 14461 heartbeater.cc:464] Registering TS with master...
I20260620 11:34:54.903460 14461 heartbeater.cc:511] Master 127.2.243.190:38067 requested a full tablet report, sending...
I20260620 11:34:54.904130 14247 ts_manager.cc:194] Registered new tserver with Master: 65dfcb6df4ba4b12a639b41c809622c9 (127.2.243.129:46749)
I20260620 11:34:54.905069 14247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:39615
WARNING: no policy specified for kudu/127.2.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
I20260620 11:34:54.929803 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.243.148:41733/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:38067
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:45709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 11:34:55 dist-test-slave-ctbw krb5kdc[13772](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781955295, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 11:34:55.060047 14467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:34:55.060400 14467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:34:55.060480 14467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:34:55.062299 14467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:34:55.062390 14467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:34:55.062455 14467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:34:55.064704 14467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:45709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.243.148:41733/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:38067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:34:55.065096 14467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:34:55.065395 14467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:34:55.072989 14477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:55.072985 14475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:55.073382 14474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:34:55.074400 14467 server_base.cc:1034] running on GCE node
I20260620 11:34:55.074618 14467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:34:55.074908 14467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:34:55.078922 14467 hybrid_clock.cc:648] HybridClock initialized: now 1781955295078892 us; error 41 us; skew 500 ppm
Jun 20 11:34:55 dist-test-slave-ctbw krb5kdc[13772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955295, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:34:55.082494 14467 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:34:55.083072 14467 webserver.cc:460] Webserver started at http://127.2.243.130:45595/ using document root <none> and password file <none>
I20260620 11:34:55.083400 14467 fs_manager.cc:359] Metadata directory not provided
I20260620 11:34:55.083513 14467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:34:55.083703 14467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 11:34:55 dist-test-slave-ctbw krb5kdc[13772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955295, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for HTTP/127.2.243.148@KRBTEST.COM
I20260620 11:34:55.100191 14467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "ea1773d9e6e94b44915d2cfccd3b442d"
format_stamp: "Formatted at 2026-06-20 11:34:55 on dist-test-slave-ctbw"
server_key: "aeb006f63541c8cd8c09729a00f33712"
server_key_iv: "e7c4e08ee03f0594a31bdeb2c9199e4e"
server_key_version: "kuduclusterkey@0"
I20260620 11:34:55.100618 14467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "ea1773d9e6e94b44915d2cfccd3b442d"
format_stamp: "Formatted at 2026-06-20 11:34:55 on dist-test-slave-ctbw"
server_key: "aeb006f63541c8cd8c09729a00f33712"
server_key_iv: "e7c4e08ee03f0594a31bdeb2c9199e4e"
server_key_version: "kuduclusterkey@0"
I20260620 11:34:55.102805 14467 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260620 11:34:55.125456 14484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:55.125705 14467 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260620 11:34:55.125797 14467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "ea1773d9e6e94b44915d2cfccd3b442d"
format_stamp: "Formatted at 2026-06-20 11:34:55 on dist-test-slave-ctbw"
server_key: "aeb006f63541c8cd8c09729a00f33712"
server_key_iv: "e7c4e08ee03f0594a31bdeb2c9199e4e"
server_key_version: "kuduclusterkey@0"
I20260620 11:34:55.125883 14467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:34:55.153811 14467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:34:55.156279 14467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:34:55.156433 14467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:34:55.156672 14467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:34:55.157040 14467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:34:55.157091 14467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:55.157183 14467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:34:55.157217 14467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:55.164533 14467 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:45957
I20260620 11:34:55.165055 14467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:34:55.165750 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 14467
I20260620 11:34:55.165843 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:34:55.166054 3022 external_mini_cluster.cc:1428] Setting key 849a2cdc1f6be2e7a62358b02ad91d38
I20260620 11:34:55.166718 14598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:45957 every 8 connection(s)
Jun 20 11:34:55 dist-test-slave-ctbw krb5kdc[13772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955295, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.131@KRBTEST.COM" created.
I20260620 11:34:55.180778 14464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:55.168730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:52421 (local address 127.2.243.190:38067)
0620 11:34:55.168823 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:55.168825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:55.168964 (+ 139us) server_negotiation.cc:407] Connection header received
0620 11:34:55.169615 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:55.169616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:55.169632 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:55.169707 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:55.170547 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.171396 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.173948 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.174282 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.174332 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:55.177116 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:34:55.177118 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:34:55.177119 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:34:55.177127 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:34:55.178987 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:34:55.179527 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:34:55.179528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:34:55.179529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:34:55.179599 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:34:55.179921 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:34:55.179923 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:34:55.179923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:34:55.180079 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:34:55.180162 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:55.180561 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 11:34:55.180657 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 11:34:55.182991 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:55.167524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:38067 (local address 127.2.243.130:52421)
0620 11:34:55.168821 (+ 1297us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:55.168870 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:55.169493 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:55.169853 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:55.169858 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:55.170121 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:55.170429 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:55.170436 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.171514 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.171515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:55.173834 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:55.173837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.174433 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.174434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:55.174577 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:55.175210 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:55.175221 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:55.176998 (+ 1777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:34:55.179095 (+ 2097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:34:55.179102 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:34:55.179105 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:34:55.179433 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:34:55.179690 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:34:55.179691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:34:55.179692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:34:55.179843 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:34:55.180193 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:34:55.180199 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:34:55.180401 (+ 202us) client_negotiation.cc:769] Sending connection context
0620 11:34:55.182517 (+ 2116us) client_negotiation.cc:240] Negotiation successful
0620 11:34:55.182705 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1247,"thread_start_us":100,"threads_started":1}
I20260620 11:34:55.183815 14599 heartbeater.cc:347] Connected to a master server at 127.2.243.190:38067
I20260620 11:34:55.183930 14599 heartbeater.cc:464] Registering TS with master...
I20260620 11:34:55.184136 14599 heartbeater.cc:511] Master 127.2.243.190:38067 requested a full tablet report, sending...
I20260620 11:34:55.184640 14247 ts_manager.cc:194] Registered new tserver with Master: ea1773d9e6e94b44915d2cfccd3b442d (127.2.243.130:45957)
I20260620 11:34:55.185153 14247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:52421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
I20260620 11:34:55.198091 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.243.148:41733/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:38067
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:45709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:34:55.300141 14604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:34:55.300438 14604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:34:55.300474 14604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:34:55.302148 14604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:34:55.302287 14604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:34:55.302402 14604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:34:55.304695 14604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:45709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.243.148:41733/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:38067
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:34:55.304955 14604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:34:55.305217 14604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:34:55.311365 14612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:55.311450 14609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:34:55.311368 14610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:34:55.314642 14604 server_base.cc:1034] running on GCE node
I20260620 11:34:55.314954 14604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:34:55.315263 14604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:34:55.316448 14604 hybrid_clock.cc:648] HybridClock initialized: now 1781955295316420 us; error 43 us; skew 500 ppm
Jun 20 11:34:55 dist-test-slave-ctbw krb5kdc[13772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955295, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:34:55.319203 14604 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:34:55.319696 14604 webserver.cc:460] Webserver started at http://127.2.243.131:45339/ using document root <none> and password file <none>
I20260620 11:34:55.319941 14604 fs_manager.cc:359] Metadata directory not provided
I20260620 11:34:55.320017 14604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:34:55.320188 14604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 11:34:55 dist-test-slave-ctbw krb5kdc[13772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955295, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for HTTP/127.2.243.148@KRBTEST.COM
I20260620 11:34:55.334453 14604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "bf795ca851fd4cc1a5697c7c28f52e85"
format_stamp: "Formatted at 2026-06-20 11:34:55 on dist-test-slave-ctbw"
server_key: "a63ace508d023dcf6045d947ab29add2"
server_key_iv: "185daf9cdc457ee083f8e8fc01b33b00"
server_key_version: "kuduclusterkey@0"
I20260620 11:34:55.334940 14604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "bf795ca851fd4cc1a5697c7c28f52e85"
format_stamp: "Formatted at 2026-06-20 11:34:55 on dist-test-slave-ctbw"
server_key: "a63ace508d023dcf6045d947ab29add2"
server_key_iv: "185daf9cdc457ee083f8e8fc01b33b00"
server_key_version: "kuduclusterkey@0"
I20260620 11:34:55.338759 14604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260620 11:34:55.351305 14619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:55.351580 14604 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 11:34:55.351848 14604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "bf795ca851fd4cc1a5697c7c28f52e85"
format_stamp: "Formatted at 2026-06-20 11:34:55 on dist-test-slave-ctbw"
server_key: "a63ace508d023dcf6045d947ab29add2"
server_key_iv: "185daf9cdc457ee083f8e8fc01b33b00"
server_key_version: "kuduclusterkey@0"
I20260620 11:34:55.351999 14604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:34:55.370504 14604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:34:55.373081 14604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:34:55.373307 14604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:34:55.373591 14604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:34:55.374022 14604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:34:55.374119 14604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:55.374229 14604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:34:55.374292 14604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:34:55.380963 14604 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:37479
I20260620 11:34:55.381508 14604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:34:55.385486 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 14604
I20260620 11:34:55.385586 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:34:55.385787 3022 external_mini_cluster.cc:1428] Setting key 8c10e47aa72817e54a6ff36d810387f8
I20260620 11:34:55.384346 14732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:37479 every 8 connection(s)
Jun 20 11:34:55 dist-test-slave-ctbw krb5kdc[13772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955295, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:34:55.395095 14464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:55.384947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:56737 (local address 127.2.243.190:38067)
0620 11:34:55.385006 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:55.385009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:55.385269 (+ 260us) server_negotiation.cc:407] Connection header received
0620 11:34:55.385994 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:55.385996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:55.386010 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:55.386045 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:55.387560 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.388070 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.389098 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.389314 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.389387 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:55.391721 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:34:55.391723 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:34:55.391723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:34:55.391731 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:34:55.393496 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:34:55.393988 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:34:55.393989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:34:55.393990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:34:55.394051 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:34:55.394325 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:34:55.394326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:34:55.394327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:34:55.394464 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:34:55.394506 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:55.394912 (+ 406us) server_negotiation.cc:299] Negotiation successful
0620 11:34:55.394995 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 11:34:55.395839 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:55.384951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:38067 (local address 127.2.243.131:56737)
0620 11:34:55.385135 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:55.385187 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:55.385863 (+ 676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:55.386814 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:55.386819 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:55.387250 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:34:55.387446 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:55.387451 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.388174 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.388175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:55.388983 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:55.388985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.389405 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.389406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:55.389528 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:55.390084 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:34:55.390092 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:34:55.391591 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:34:55.393598 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:34:55.393604 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:34:55.393607 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:34:55.393897 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:34:55.394121 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:34:55.394121 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:34:55.394122 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:34:55.394251 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:34:55.394551 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:34:55.394557 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:34:55.394729 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 11:34:55.395417 (+ 688us) client_negotiation.cc:240] Negotiation successful
0620 11:34:55.395584 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":111,"threads_started":1}
I20260620 11:34:55.396526 14733 heartbeater.cc:347] Connected to a master server at 127.2.243.190:38067
I20260620 11:34:55.396624 14733 heartbeater.cc:464] Registering TS with master...
I20260620 11:34:55.396852 14733 heartbeater.cc:511] Master 127.2.243.190:38067 requested a full tablet report, sending...
I20260620 11:34:55.397612 14247 ts_manager.cc:194] Registered new tserver with Master: bf795ca851fd4cc1a5697c7c28f52e85 (127.2.243.131:37479)
I20260620 11:34:55.398144 14247 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:56737
I20260620 11:34:55.399363 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:34:55.411734 14464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:55.403194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36010 (local address 127.2.243.190:38067)
0620 11:34:55.403240 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:55.403243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:55.403587 (+ 344us) server_negotiation.cc:407] Connection header received
0620 11:34:55.403687 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:55.403689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:55.403700 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:55.403731 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:34:55.404600 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.405364 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.406481 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.406762 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.406821 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:55.408539 (+ 1718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:34:55.408540 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:34:55.408541 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:34:55.408549 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:34:55.410367 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:34:55.410789 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:34:55.410790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:34:55.410791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:34:55.410886 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:34:55.411113 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:34:55.411114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:34:55.411115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:34:55.411252 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:34:55.411335 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:55.411554 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 11:34:55.411634 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:34:55.416766 14247 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:34:55.419196 14247 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:34:55.438724 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:55.422835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:37479 (local address 127.0.0.1:60280)
0620 11:34:55.424594 (+ 1759us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:55.424639 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:55.424708 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:55.426545 (+ 1837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:55.426546 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:55.426547 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:34:55.426575 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:55.426576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.432106 (+ 5530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.432107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:55.433340 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:55.433344 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.438360 (+ 5016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.438362 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:55.438462 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:55.438478 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 11:34:55.438557 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 11:34:55.438618 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1721,"thread_start_us":109,"threads_started":1}
I20260620 11:34:55.439733 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:55.423120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:46749 (local address 127.0.0.1:37546)
0620 11:34:55.425241 (+ 2121us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:55.425251 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:55.425304 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:55.426380 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:55.426385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:55.426388 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:34:55.426471 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:55.426473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.428685 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.428685 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:55.433865 (+ 5180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:55.433868 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.439422 (+ 5554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.439423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:55.439503 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:55.439508 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:34:55.439572 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 11:34:55.439627 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2104,"thread_start_us":69,"threads_started":1}
I20260620 11:34:55.441529 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:55.423651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37546 (local address 127.2.243.129:46749)
0620 11:34:55.425186 (+ 1535us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:55.425191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:55.425882 (+ 691us) server_negotiation.cc:407] Connection header received
0620 11:34:55.425921 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:55.425923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:55.425940 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:55.426029 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:34:55.426643 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.427752 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.436262 (+ 8510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.437113 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.437198 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:55.437275 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:55.441354 (+ 4079us) server_negotiation.cc:299] Negotiation successful
0620 11:34:55.441446 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1502,"thread_start_us":64,"threads_started":1}
I20260620 11:34:55.442014 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:55.423428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60280 (local address 127.2.243.131:37479)
0620 11:34:55.424868 (+ 1440us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:55.424873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:55.424884 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:34:55.424923 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:55.424925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:55.424942 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:55.425062 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:34:55.428021 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.431998 (+ 3977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.437340 (+ 5342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.438160 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.438239 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:55.438319 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:55.441880 (+ 3561us) server_negotiation.cc:299] Negotiation successful
0620 11:34:55.441931 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1402,"thread_start_us":89,"threads_started":1}
I20260620 11:34:55.442924 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:55.423957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:45957 (local address 127.0.0.1:43258)
0620 11:34:55.427894 (+ 3937us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:55.427905 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:55.427950 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:55.429297 (+ 1347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:55.429298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:55.429300 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:34:55.429343 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:55.429344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.434566 (+ 5222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.434567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:55.436146 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:55.436149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.442648 (+ 6499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.442649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:55.442716 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:55.442721 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:34:55.442771 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 11:34:55.442829 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3919,"thread_start_us":70,"threads_started":1}
I20260620 11:34:55.442955 14395 tablet_service.cc:1467] Processing CreateTablet for tablet 7259fecf6e164880b5891aab829e9267 (DEFAULT_TABLE table=test-table [id=884e92330f564d319dd4cf5ae9a90527]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:34:55.443403 14395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7259fecf6e164880b5891aab829e9267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:34:55.443862 14667 tablet_service.cc:1467] Processing CreateTablet for tablet 7259fecf6e164880b5891aab829e9267 (DEFAULT_TABLE table=test-table [id=884e92330f564d319dd4cf5ae9a90527]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:34:55.444212 14667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7259fecf6e164880b5891aab829e9267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:34:55.448391 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:55.424213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43258 (local address 127.2.243.130:45957)
0620 11:34:55.425851 (+ 1638us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:55.425856 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:55.429050 (+ 3194us) server_negotiation.cc:407] Connection header received
0620 11:34:55.429105 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:55.429108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:55.429126 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:55.429208 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:34:55.430527 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.434466 (+ 3939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.439850 (+ 5384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.442463 (+ 2613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.442535 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:55.442612 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:55.448235 (+ 5623us) server_negotiation.cc:299] Negotiation successful
0620 11:34:55.448297 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1175,"thread_start_us":76,"threads_started":1}
I20260620 11:34:55.449193 14515 tablet_service.cc:1467] Processing CreateTablet for tablet 7259fecf6e164880b5891aab829e9267 (DEFAULT_TABLE table=test-table [id=884e92330f564d319dd4cf5ae9a90527]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:34:55.449538 14515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7259fecf6e164880b5891aab829e9267. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:34:55.453917 14754 tablet_bootstrap.cc:492] T 7259fecf6e164880b5891aab829e9267 P ea1773d9e6e94b44915d2cfccd3b442d: Bootstrap starting.
I20260620 11:34:55.455101 14754 tablet_bootstrap.cc:654] T 7259fecf6e164880b5891aab829e9267 P ea1773d9e6e94b44915d2cfccd3b442d: Neither blocks nor log segments found. Creating new log.
I20260620 11:34:55.455335 14753 tablet_bootstrap.cc:492] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85: Bootstrap starting.
I20260620 11:34:55.455441 14752 tablet_bootstrap.cc:492] T 7259fecf6e164880b5891aab829e9267 P 65dfcb6df4ba4b12a639b41c809622c9: Bootstrap starting.
I20260620 11:34:55.456228 14752 tablet_bootstrap.cc:654] T 7259fecf6e164880b5891aab829e9267 P 65dfcb6df4ba4b12a639b41c809622c9: Neither blocks nor log segments found. Creating new log.
I20260620 11:34:55.456471 14754 log.cc:824] T 7259fecf6e164880b5891aab829e9267 P ea1773d9e6e94b44915d2cfccd3b442d: Log is configured to *not* fsync() on all Append() calls
I20260620 11:34:55.456621 14752 log.cc:824] T 7259fecf6e164880b5891aab829e9267 P 65dfcb6df4ba4b12a639b41c809622c9: Log is configured to *not* fsync() on all Append() calls
I20260620 11:34:55.457152 14753 tablet_bootstrap.cc:654] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85: Neither blocks nor log segments found. Creating new log.
I20260620 11:34:55.457481 14753 log.cc:824] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85: Log is configured to *not* fsync() on all Append() calls
I20260620 11:34:55.458233 14754 tablet_bootstrap.cc:492] T 7259fecf6e164880b5891aab829e9267 P ea1773d9e6e94b44915d2cfccd3b442d: No bootstrap required, opened a new log
I20260620 11:34:55.458308 14754 ts_tablet_manager.cc:1397] T 7259fecf6e164880b5891aab829e9267 P ea1773d9e6e94b44915d2cfccd3b442d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 11:34:55.460323 14754 raft_consensus.cc:348] T 7259fecf6e164880b5891aab829e9267 P ea1773d9e6e94b44915d2cfccd3b442d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1773d9e6e94b44915d2cfccd3b442d" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 45957 } } peers { permanent_uuid: "bf795ca851fd4cc1a5697c7c28f52e85" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 37479 } } peers { permanent_uuid: "65dfcb6df4ba4b12a639b41c809622c9" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 46749 } }
I20260620 11:34:55.460501 14754 raft_consensus.cc:374] T 7259fecf6e164880b5891aab829e9267 P ea1773d9e6e94b44915d2cfccd3b442d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:34:55.460541 14754 raft_consensus.cc:729] T 7259fecf6e164880b5891aab829e9267 P ea1773d9e6e94b44915d2cfccd3b442d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea1773d9e6e94b44915d2cfccd3b442d, State: Initialized, Role: FOLLOWER
I20260620 11:34:55.460680 14754 consensus_queue.cc:260] T 7259fecf6e164880b5891aab829e9267 P ea1773d9e6e94b44915d2cfccd3b442d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1773d9e6e94b44915d2cfccd3b442d" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 45957 } } peers { permanent_uuid: "bf795ca851fd4cc1a5697c7c28f52e85" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 37479 } } peers { permanent_uuid: "65dfcb6df4ba4b12a639b41c809622c9" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 46749 } }
I20260620 11:34:55.460943 14754 ts_tablet_manager.cc:1428] T 7259fecf6e164880b5891aab829e9267 P ea1773d9e6e94b44915d2cfccd3b442d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:34:55.461055 14599 heartbeater.cc:503] Master 127.2.243.190:38067 was elected leader, sending a full tablet report...
I20260620 11:34:55.464247 14753 tablet_bootstrap.cc:492] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85: No bootstrap required, opened a new log
I20260620 11:34:55.464332 14753 ts_tablet_manager.cc:1397] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260620 11:34:55.466986 14753 raft_consensus.cc:348] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1773d9e6e94b44915d2cfccd3b442d" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 45957 } } peers { permanent_uuid: "bf795ca851fd4cc1a5697c7c28f52e85" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 37479 } } peers { permanent_uuid: "65dfcb6df4ba4b12a639b41c809622c9" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 46749 } }
I20260620 11:34:55.467183 14753 raft_consensus.cc:374] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:34:55.467278 14753 raft_consensus.cc:729] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf795ca851fd4cc1a5697c7c28f52e85, State: Initialized, Role: FOLLOWER
I20260620 11:34:55.467432 14753 consensus_queue.cc:260] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1773d9e6e94b44915d2cfccd3b442d" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 45957 } } peers { permanent_uuid: "bf795ca851fd4cc1a5697c7c28f52e85" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 37479 } } peers { permanent_uuid: "65dfcb6df4ba4b12a639b41c809622c9" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 46749 } }
I20260620 11:34:55.467716 14753 ts_tablet_manager.cc:1428] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 11:34:55.467861 14733 heartbeater.cc:503] Master 127.2.243.190:38067 was elected leader, sending a full tablet report...
I20260620 11:34:55.469022 14752 tablet_bootstrap.cc:492] T 7259fecf6e164880b5891aab829e9267 P 65dfcb6df4ba4b12a639b41c809622c9: No bootstrap required, opened a new log
I20260620 11:34:55.469153 14752 ts_tablet_manager.cc:1397] T 7259fecf6e164880b5891aab829e9267 P 65dfcb6df4ba4b12a639b41c809622c9: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260620 11:34:55.472450 14752 raft_consensus.cc:348] T 7259fecf6e164880b5891aab829e9267 P 65dfcb6df4ba4b12a639b41c809622c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1773d9e6e94b44915d2cfccd3b442d" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 45957 } } peers { permanent_uuid: "bf795ca851fd4cc1a5697c7c28f52e85" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 37479 } } peers { permanent_uuid: "65dfcb6df4ba4b12a639b41c809622c9" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 46749 } }
I20260620 11:34:55.473332 14752 raft_consensus.cc:374] T 7259fecf6e164880b5891aab829e9267 P 65dfcb6df4ba4b12a639b41c809622c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:34:55.473412 14752 raft_consensus.cc:729] T 7259fecf6e164880b5891aab829e9267 P 65dfcb6df4ba4b12a639b41c809622c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65dfcb6df4ba4b12a639b41c809622c9, State: Initialized, Role: FOLLOWER
I20260620 11:34:55.473592 14752 consensus_queue.cc:260] T 7259fecf6e164880b5891aab829e9267 P 65dfcb6df4ba4b12a639b41c809622c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1773d9e6e94b44915d2cfccd3b442d" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 45957 } } peers { permanent_uuid: "bf795ca851fd4cc1a5697c7c28f52e85" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 37479 } } peers { permanent_uuid: "65dfcb6df4ba4b12a639b41c809622c9" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 46749 } }
I20260620 11:34:55.473924 14752 ts_tablet_manager.cc:1428] T 7259fecf6e164880b5891aab829e9267 P 65dfcb6df4ba4b12a639b41c809622c9: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260620 11:34:55.474087 14461 heartbeater.cc:503] Master 127.2.243.190:38067 was elected leader, sending a full tablet report...
I20260620 11:34:55.474639 14759 raft_consensus.cc:482] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:34:55.474727 14759 raft_consensus.cc:504] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1773d9e6e94b44915d2cfccd3b442d" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 45957 } } peers { permanent_uuid: "bf795ca851fd4cc1a5697c7c28f52e85" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 37479 } } peers { permanent_uuid: "65dfcb6df4ba4b12a639b41c809622c9" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 46749 } }
I20260620 11:34:55.477824 14759 leader_election.cc:288] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ea1773d9e6e94b44915d2cfccd3b442d (127.2.243.130:45957), 65dfcb6df4ba4b12a639b41c809622c9 (127.2.243.129:46749)
I20260620 11:34:55.478972 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:55.475381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:55805 (local address 127.2.243.130:45957)
0620 11:34:55.475415 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:55.475416 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:55.475423 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:34:55.475444 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:55.475445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:55.475453 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:55.475491 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:34:55.475654 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.476490 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.477854 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.478369 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.478435 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:55.478443 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:55.478875 (+ 432us) server_negotiation.cc:299] Negotiation successful
0620 11:34:55.478914 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 11:34:55.478713 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:55.475164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:45957 (local address 127.2.243.131:55805)
0620 11:34:55.475267 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:55.475278 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:55.475313 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:55.475531 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:55.475532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:55.475533 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:34:55.475596 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:55.475598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.476615 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.476616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:55.477752 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:55.477755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.478474 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.478475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:55.478554 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:55.478557 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:34:55.478598 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 11:34:55.478636 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 11:34:55.479507 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7259fecf6e164880b5891aab829e9267" candidate_uuid: "bf795ca851fd4cc1a5697c7c28f52e85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea1773d9e6e94b44915d2cfccd3b442d" is_pre_election: true
I20260620 11:34:55.479677 14553 raft_consensus.cc:2454] T 7259fecf6e164880b5891aab829e9267 P ea1773d9e6e94b44915d2cfccd3b442d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf795ca851fd4cc1a5697c7c28f52e85 in term 0.
I20260620 11:34:55.480402 14622 leader_election.cc:302] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf795ca851fd4cc1a5697c7c28f52e85, ea1773d9e6e94b44915d2cfccd3b442d; no voters:
I20260620 11:34:55.482432 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:55.477598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:46749 (local address 127.2.243.131:33635)
0620 11:34:55.479128 (+ 1530us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:55.479138 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:34:55.479185 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:34:55.479391 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:34:55.479392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:34:55.479393 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:34:55.479424 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:55.479426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.480530 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.480531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:55.481522 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:34:55.481524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.482246 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.482247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:34:55.482299 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:55.482302 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:34:55.482342 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 11:34:55.482371 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1518,"thread_start_us":90,"threads_started":1}
I20260620 11:34:55.482656 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:55.477767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:33635 (local address 127.2.243.129:46749)
0620 11:34:55.477806 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:55.477808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:55.479239 (+ 1431us) server_negotiation.cc:407] Connection header received
0620 11:34:55.479268 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:55.479269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:55.479280 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:55.479344 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:34:55.479496 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.480201 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.481614 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.482145 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.482210 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:55.482218 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:55.482567 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 11:34:55.482604 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:34:55.482895 14403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7259fecf6e164880b5891aab829e9267" candidate_uuid: "bf795ca851fd4cc1a5697c7c28f52e85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65dfcb6df4ba4b12a639b41c809622c9" is_pre_election: true
I20260620 11:34:55.483006 14403 raft_consensus.cc:2454] T 7259fecf6e164880b5891aab829e9267 P 65dfcb6df4ba4b12a639b41c809622c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bf795ca851fd4cc1a5697c7c28f52e85 in term 0.
I20260620 11:34:55.483363 14759 raft_consensus.cc:2781] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:34:55.483484 14759 raft_consensus.cc:482] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:34:55.483543 14759 raft_consensus.cc:3037] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:34:55.484416 14759 raft_consensus.cc:504] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1773d9e6e94b44915d2cfccd3b442d" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 45957 } } peers { permanent_uuid: "bf795ca851fd4cc1a5697c7c28f52e85" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 37479 } } peers { permanent_uuid: "65dfcb6df4ba4b12a639b41c809622c9" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 46749 } }
I20260620 11:34:55.484767 14553 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7259fecf6e164880b5891aab829e9267" candidate_uuid: "bf795ca851fd4cc1a5697c7c28f52e85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea1773d9e6e94b44915d2cfccd3b442d"
I20260620 11:34:55.484858 14553 raft_consensus.cc:3037] T 7259fecf6e164880b5891aab829e9267 P ea1773d9e6e94b44915d2cfccd3b442d [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:34:55.485654 14553 raft_consensus.cc:2454] T 7259fecf6e164880b5891aab829e9267 P ea1773d9e6e94b44915d2cfccd3b442d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf795ca851fd4cc1a5697c7c28f52e85 in term 1.
I20260620 11:34:55.486235 14622 leader_election.cc:302] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bf795ca851fd4cc1a5697c7c28f52e85, ea1773d9e6e94b44915d2cfccd3b442d; no voters:
I20260620 11:34:55.486562 14403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7259fecf6e164880b5891aab829e9267" candidate_uuid: "bf795ca851fd4cc1a5697c7c28f52e85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65dfcb6df4ba4b12a639b41c809622c9"
I20260620 11:34:55.486629 14403 raft_consensus.cc:3037] T 7259fecf6e164880b5891aab829e9267 P 65dfcb6df4ba4b12a639b41c809622c9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:34:55.487429 14403 raft_consensus.cc:2454] T 7259fecf6e164880b5891aab829e9267 P 65dfcb6df4ba4b12a639b41c809622c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bf795ca851fd4cc1a5697c7c28f52e85 in term 1.
I20260620 11:34:55.487638 14759 leader_election.cc:288] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85 [CANDIDATE]: Term 1 election: Requested vote from peers ea1773d9e6e94b44915d2cfccd3b442d (127.2.243.130:45957), 65dfcb6df4ba4b12a639b41c809622c9 (127.2.243.129:46749)
I20260620 11:34:55.487746 14759 raft_consensus.cc:2781] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:34:55.487826 14759 raft_consensus.cc:686] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85 [term 1 LEADER]: Becoming Leader. State: Replica: bf795ca851fd4cc1a5697c7c28f52e85, State: Running, Role: LEADER
I20260620 11:34:55.487946 14759 consensus_queue.cc:237] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1773d9e6e94b44915d2cfccd3b442d" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 45957 } } peers { permanent_uuid: "bf795ca851fd4cc1a5697c7c28f52e85" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 37479 } } peers { permanent_uuid: "65dfcb6df4ba4b12a639b41c809622c9" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 46749 } }
I20260620 11:34:55.489029 14247 catalog_manager.cc:5495] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85 reported cstate change: term changed from 0 to 1, leader changed from <none> to bf795ca851fd4cc1a5697c7c28f52e85 (127.2.243.131). New cstate: current_term: 1 leader_uuid: "bf795ca851fd4cc1a5697c7c28f52e85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea1773d9e6e94b44915d2cfccd3b442d" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 45957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bf795ca851fd4cc1a5697c7c28f52e85" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 37479 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "65dfcb6df4ba4b12a639b41c809622c9" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 46749 } health_report { overall_health: UNKNOWN } } }
I20260620 11:34:55.509182 14748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:55.503379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60294 (local address 127.2.243.131:37479)
0620 11:34:55.503420 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 11:34:55.503422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:34:55.503438 (+ 16us) server_negotiation.cc:407] Connection header received
0620 11:34:55.503482 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:34:55.503483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:34:55.503505 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:34:55.503568 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:34:55.504697 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.505393 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.507066 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:34:55.507339 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:34:55.507404 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:34:55.507609 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:34:55.507750 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:34:55.507819 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 11:34:55.509059 (+ 1240us) server_negotiation.cc:299] Negotiation successful
0620 11:34:55.509082 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:34:55.511970 14403 raft_consensus.cc:1261] T 7259fecf6e164880b5891aab829e9267 P 65dfcb6df4ba4b12a639b41c809622c9 [term 1 FOLLOWER]: Refusing update from remote peer bf795ca851fd4cc1a5697c7c28f52e85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:34:55.512431 14762 consensus_queue.cc:1035] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65dfcb6df4ba4b12a639b41c809622c9" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 46749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:34:55.514731 14553 raft_consensus.cc:1261] T 7259fecf6e164880b5891aab829e9267 P ea1773d9e6e94b44915d2cfccd3b442d [term 1 FOLLOWER]: Refusing update from remote peer bf795ca851fd4cc1a5697c7c28f52e85: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:34:55.515076 14762 consensus_queue.cc:1035] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea1773d9e6e94b44915d2cfccd3b442d" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 45957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:34:55.521725 14765 mvcc.cc:205] Tried to move back new op lower bound from 7298888890413899776 to 7298888890320289792. Current Snapshot: MvccSnapshot[applied={T|T < 7298888890413899776}]
I20260620 11:34:55.521891 14766 mvcc.cc:205] Tried to move back new op lower bound from 7298888890413899776 to 7298888890320289792. Current Snapshot: MvccSnapshot[applied={T|T < 7298888890413899776}]
I20260620 11:34:55.525988 14247 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:34:55.526078 14247 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:34:55.526942 14247 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a045875e5d904a26b3d977b164dc5abd: Sending DeleteTablet for 3 replicas of tablet 7259fecf6e164880b5891aab829e9267
I20260620 11:34:55.527277 14395 tablet_service.cc:1514] Processing DeleteTablet for tablet 7259fecf6e164880b5891aab829e9267 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:34:55 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:37546
I20260620 11:34:55.527664 14667 tablet_service.cc:1514] Processing DeleteTablet for tablet 7259fecf6e164880b5891aab829e9267 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:34:55 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:60280
I20260620 11:34:55.528008 14515 tablet_service.cc:1514] Processing DeleteTablet for tablet 7259fecf6e164880b5891aab829e9267 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:34:55 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:43258
I20260620 11:34:55.528568 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 14330
I20260620 11:34:55.534055 14776 tablet_replica.cc:330] T 7259fecf6e164880b5891aab829e9267 P ea1773d9e6e94b44915d2cfccd3b442d: stopping tablet replica
I20260620 11:34:55.534184 14776 raft_consensus.cc:2229] T 7259fecf6e164880b5891aab829e9267 P ea1773d9e6e94b44915d2cfccd3b442d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:34:55.534314 14776 raft_consensus.cc:2258] T 7259fecf6e164880b5891aab829e9267 P ea1773d9e6e94b44915d2cfccd3b442d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:34:55.534476 14775 tablet_replica.cc:330] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85: stopping tablet replica
I20260620 11:34:55.534575 14775 raft_consensus.cc:2229] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:34:55.534677 14775 raft_consensus.cc:2258] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:34:55.535283 14775 ts_tablet_manager.cc:1905] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 11:34:55.536001 14235 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.243.129:46749 (error 108)
I20260620 11:34:55.536839 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:34:55.536736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:46749 (local address 127.0.0.1:37560)
0620 11:34:55.536780 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 11:34:55.536808 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.129:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 11:34:55.536940 14775 ts_tablet_manager.cc:1918] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:34:55.537017 14775 log.cc:1196] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/7259fecf6e164880b5891aab829e9267
W20260620 11:34:55.537032 14235 catalog_manager.cc:4584] TS 65dfcb6df4ba4b12a639b41c809622c9 (127.2.243.129:46749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7259fecf6e164880b5891aab829e9267: Network error: Client connection negotiation failed: client connection to 127.2.243.129:46749: connect: Connection refused (error 111)
I20260620 11:34:55.537282 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 14467
I20260620 11:34:55.537293 14775 ts_tablet_manager.cc:1939] T 7259fecf6e164880b5891aab829e9267 P bf795ca851fd4cc1a5697c7c28f52e85: Deleting consensus metadata
I20260620 11:34:55.538057 14233 catalog_manager.cc:4841] TS bf795ca851fd4cc1a5697c7c28f52e85 (127.2.243.131:37479): tablet 7259fecf6e164880b5891aab829e9267 (table test-table [id=884e92330f564d319dd4cf5ae9a90527]) successfully deleted
I20260620 11:34:55.543110 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 14604
I20260620 11:34:55.550628 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 14216
I20260620 11:34:55.555840 3022 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 11:34:55.890156 3022 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 11:34:55.890285 3022 mini_ranger.cc:66] Stopping Ranger...
I20260620 11:34:56.235335 3022 mini_ranger.cc:68] Stopped Ranger
2026-06-20 11:34:56.236 UTC [13802] LOG: received smart shutdown request
2026-06-20 11:34:56.239 UTC [13802] LOG: background worker "logical replication launcher" (PID 13811) exited with exit code 1
2026-06-20 11:34:56.239 UTC [13806] LOG: shutting down
2026-06-20 11:34:56.391 UTC [13802] LOG: database system is shut down
2026-06-20T11:34:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46036 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:34:56 dist-test-slave-ctbw krb5kdc[14786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:34:56 dist-test-slave-ctbw krb5kdc[14786](info): set up 2 sockets
Jun 20 11:34:56 dist-test-slave-ctbw krb5kdc[14786](info): commencing operation
krb5kdc: starting...
W20260620 11:34:57.543370 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:34:57 dist-test-slave-ctbw krb5kdc[14786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:34:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:34:57Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.2.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.2.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.2.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/rangeradmin_127.2.243.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.2.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/rangeradmin_127.2.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.2.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.2.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.2.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/rangerlookup_127.2.243.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.2.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/rangerlookup_127.2.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.2.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/HTTP_127.2.243.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/HTTP_127.2.243.148@KRBTEST.COM.keytab.
I20260620 11:34:57.711601 3022 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskDepOyZ/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/postgres -l logfile start
2026-06-20 11:35:00.178 UTC [14816] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 11:35:00.178 UTC [14816] LOG: listening on IPv4 address "127.2.243.148", port 46933
2026-06-20 11:35:00.181 UTC [14816] LOG: listening on Unix socket "/tmp/.s.PGSQL.46933"
2026-06-20 11:35:00.203 UTC [14819] LOG: database system was shut down at 2026-06-20 11:34:58 UTC
2026-06-20 11:35:00.207 UTC [14816] LOG: database system is ready to accept connections
I20260620 11:35:01.154992 3022 mini_postgres.cc:96] Postgres bound to 46933
2026-06-20 11:35:01.161 UTC [14827] FATAL: database "slave" does not exist
127.2.243.148:46933 - accepting connections
I20260620 11:35:01.161623 3022 mini_ranger.cc:161] Starting Ranger...
I20260620 11:35:01.176736 3022 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 11:35:01.645211 3022 mini_ranger.cc:87] Created ranger Postgres database
I20260620 11:35:01.645295 3022 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ranger-admin
2026-06-20 11:35:01,925 [I] DB FLAVOR :POSTGRES
2026-06-20 11:35:01,925 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 11:35:01,925 [I] Checking connection..
2026-06-20 11:35:01,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 11:35:02,184 [I] Checking connection passed.
2026-06-20 11:35:02,184 [I] --------- Verifying version history table ---------
2026-06-20 11:35:02,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 11:35:02,426 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 11:35:02,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 11:35:02,674 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 11:35:02,675 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 11:35:02,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 11:35:02.920 UTC [14891] WARNING: there is no transaction in progress
2026-06-20 11:35:02,926 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 11:35:02,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 11:35:03,173 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 11:35:03,173 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 11:35:03,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 11:35:03,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 11:35:03,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-ctbw.c.gcp-upstream.internal','N') ;"
2026-06-20 11:35:03,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 11:35:04,163 [I] Table x_portal_user does not exist in database ranger
2026-06-20 11:35:04,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 11:35:04,401 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 11:35:04,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 11:35:04,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 11:35:04,871 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 11:35:04,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 11:35:05.859 UTC [14999] WARNING: there is no transaction in progress
2026-06-20 11:35:05.875 UTC [14999] WARNING: there is no transaction in progress
2026-06-20 11:35:05.892 UTC [14999] WARNING: there is no transaction in progress
2026-06-20 11:35:05.908 UTC [14999] WARNING: there is no transaction in progress
2026-06-20 11:35:05.924 UTC [14999] WARNING: there is no transaction in progress
2026-06-20 11:35:05.945 UTC [14999] WARNING: there is no transaction in progress
2026-06-20 11:35:06.012 UTC [14999] WARNING: there is no transaction in progress
2026-06-20 11:35:06.019 UTC [14999] WARNING: there is no transaction in progress
2026-06-20 11:35:06.032 UTC [14999] WARNING: there is no transaction in progress
2026-06-20 11:35:06.042 UTC [14999] WARNING: there is no transaction in progress
2026-06-20 11:35:06.052 UTC [14999] WARNING: there is no transaction in progress
2026-06-20 11:35:06.065 UTC [14999] WARNING: there is no transaction in progress
2026-06-20 11:35:06.075 UTC [14999] WARNING: there is no transaction in progress
2026-06-20 11:35:06.081 UTC [14999] WARNING: there is no transaction in progress
2026-06-20 11:35:06.087 UTC [14999] WARNING: there is no transaction in progress
2026-06-20 11:35:06.095 UTC [14999] WARNING: there is no transaction in progress
2026-06-20 11:35:06,712 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 11:35:06,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 11:35:06,969 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 11:35:06,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 11:35:07,231 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 11:35:07,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 11:35:07,465 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 11:35:07,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 11:35:07,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 11:35:07,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-ctbw.c.gcp-upstream.internal';"
2026-06-20 11:35:08,218 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 11:35:08,219 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 11:35:08,470 [I] DB_PATCHES have already been applied
I20260620 11:35:08.476295 3022 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ranger-admin:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskDepOyZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 11:35:08.476363 3022 mini_ranger.cc:193] Using host: 127.2.243.148
I20260620 11:35:08.479573 3022 mini_ranger.cc:227] Ranger admin URL: http://127.2.243.148:47449
Jun 20, 2026 11:35:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:35:08 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:35:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 11:35:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 11:35:08 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 11:35:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 11:35:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 11:35:09.564409 3022 mini_ranger.cc:160] Time spent starting Ranger: real 8.403s user 0.000s sys 0.009s
Jun 20, 2026 11:35:29 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 11:35:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 11:35:29 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 11:35:29 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 11:35:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 11:35:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 11:35:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 11:35:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 11:35:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 11:35:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 11:35:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 11:35:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 11:35:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 11:35:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 11:35:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 11:35:29 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 11:35:29 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 11:35:30 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 11:35:31.399122 3022 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.2.243.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.2.243.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.2.243.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/rangerkms_127.2.243.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.2.243.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/rangerkms_127.2.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.2.243.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.2.243.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.2.243.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/HTTP_127.2.243.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.2.243.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/HTTP_127.2.243.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 11:35:31 dist-test-slave-ctbw krb5kdc[14786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955331, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 11:35:31.480921 3022 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 11:35:31.490373 3022 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 11:35:32.362700 3022 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 11:35:32.362789 3022 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ranger-kms
I20260620 11:35:32.362807 3022 mini_ranger_kms.cc:227] Using postgres at 127.2.243.148:46933
2026-06-20 11:35:32,527 [I] DB FLAVOR :POSTGRES
2026-06-20 11:35:32,527 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 11:35:32,527 [I] Checking connection
2026-06-20 11:35:32,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 11:35:32,760 [I] connection success
2026-06-20 11:35:32,761 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 11:35:32,761 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 11:35:32,761 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 11:35:33,001 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 11:35:33,001 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 11:35:33,001 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 11:35:33,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.2.243.148:46933/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 11:35:33,267 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 11:35:33.737227 3022 mini_ranger_kms.cc:310] Created kms service
I20260620 11:35:33.959558 3022 mini_ranger_kms.cc:326] Created kudu user
I20260620 11:35:34.006470 3022 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 11:35:34 dist-test-slave-ctbw krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955331, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.2.243.148@KRBTEST.COM
I20260620 11:35:34.434700 3022 mini_ranger_kms.cc:384] Added ranger policy
I20260620 11:35:34.434863 3022 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ranger-kms:/tmp/dist-test-taskDepOyZ/build/release/bin/postgresql.jar:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskDepOyZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 11:35:34.434917 3022 mini_ranger_kms.cc:240] Using host: 127.2.243.148
I20260620 11:35:34.438534 3022 mini_ranger_kms.cc:276] Ranger KMS PID: 15179
I20260620 11:35:34.438620 3022 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.2.243.148:49001
Jun 20, 2026 11:35:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:35:35 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:35:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 11:35:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 11:35:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 11:35:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 11:35:36 dist-test-slave-ctbw krb5kdc[14786](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781955336, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.243.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:36.731501 3022 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.251s user 0.004s sys 0.143s
I20260620 11:35:36.731609 3022 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 11:35:36.731660 3022 mini_ranger_kms.cc:401] 127.2.243.148:49001/kms/v1/keys
Jun 20 11:35:37 dist-test-slave-ctbw krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955336, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.243.148@KRBTEST.COM for HTTP/127.2.243.148@KRBTEST.COM
Jun 20 11:35:37 dist-test-slave-ctbw krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955336, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.2.243.148@KRBTEST.COM for HTTP/127.2.243.148@KRBTEST.COM
Jun 20 11:35:39 dist-test-slave-ctbw krb5kdc[14786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:35:39.749140 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:33281
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:33281
--ranger_config_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.243.148:49001/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:35:39.841190 15230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:39.841401 15230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:39.841423 15230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:39.842595 15230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:35:39.842638 15230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:39.842653 15230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:39.842665 15230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:35:39.842677 15230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:35:39.844403 15230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/wal
--ranger_kms_url=127.2.243.148:49001/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:33281
--ranger_config_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:33281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:39.844640 15230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:39.844901 15230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:39.850322 15236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:39.850363 15235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:39.850522 15230 server_base.cc:1034] running on GCE node
W20260620 11:35:39.850700 15238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:39.851008 15230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:39.851348 15230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:39.852522 15230 hybrid_clock.cc:648] HybridClock initialized: now 1781955339852507 us; error 37 us; skew 500 ppm
Jun 20 11:35:39 dist-test-slave-ctbw krb5kdc[14786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955339, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:39.859866 15230 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:35:39.860473 15230 webserver.cc:460] Webserver started at http://127.2.243.190:39343/ using document root <none> and password file <none>
I20260620 11:35:39.860743 15230 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:39.860798 15230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:35:39.860932 15230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 11:35:39 dist-test-slave-ctbw krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955339, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for HTTP/127.2.243.148@KRBTEST.COM
I20260620 11:35:39.916885 15230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "2b797c0520f445e285508efbc339532f"
format_stamp: "Formatted at 2026-06-20 11:35:39 on dist-test-slave-ctbw"
server_key: "5c5f1cdd794570eeac139b85f8b92296"
server_key_iv: "9e4e64390669edb186d92c9f0b0ebd3b"
server_key_version: "kuduclusterkey@0"
I20260620 11:35:39.917338 15230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "2b797c0520f445e285508efbc339532f"
format_stamp: "Formatted at 2026-06-20 11:35:39 on dist-test-slave-ctbw"
server_key: "5c5f1cdd794570eeac139b85f8b92296"
server_key_iv: "9e4e64390669edb186d92c9f0b0ebd3b"
server_key_version: "kuduclusterkey@0"
I20260620 11:35:39.918993 15230 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 11:35:39.934405 15245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:39.934680 15230 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:35:39.934789 15230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "2b797c0520f445e285508efbc339532f"
format_stamp: "Formatted at 2026-06-20 11:35:39 on dist-test-slave-ctbw"
server_key: "5c5f1cdd794570eeac139b85f8b92296"
server_key_iv: "9e4e64390669edb186d92c9f0b0ebd3b"
server_key_version: "kuduclusterkey@0"
I20260620 11:35:39.934953 15230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:39.944754 15230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:39.947055 15230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:39.947222 15230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:39.952189 15230 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:33281
I20260620 11:35:39.952216 15297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:33281 every 8 connection(s)
I20260620 11:35:39.952606 15230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:35:39.953401 15298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:35:39.955854 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 15230
I20260620 11:35:39.955974 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:35:39.956154 3022 external_mini_cluster.cc:1428] Setting key 767536f7536f5ac48639b1afd29308bc
I20260620 11:35:39.956489 15298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f: Bootstrap starting.
I20260620 11:35:39.957226 15298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f: Neither blocks nor log segments found. Creating new log.
I20260620 11:35:39.957553 15298 log.cc:824] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f: Log is configured to *not* fsync() on all Append() calls
I20260620 11:35:39.958379 15298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f: No bootstrap required, opened a new log
I20260620 11:35:39.960505 15298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b797c0520f445e285508efbc339532f" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33281 } }
I20260620 11:35:39.960695 15298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:35:39.960731 15298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b797c0520f445e285508efbc339532f, State: Initialized, Role: FOLLOWER
I20260620 11:35:39.960860 15298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b797c0520f445e285508efbc339532f" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33281 } }
I20260620 11:35:39.960932 15298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:35:39.960963 15298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:35:39.961005 15298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:35:39.961580 15298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b797c0520f445e285508efbc339532f" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33281 } }
I20260620 11:35:39.961663 15298 leader_election.cc:302] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b797c0520f445e285508efbc339532f; no voters:
I20260620 11:35:39.961815 15298 leader_election.cc:288] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:35:39.962106 15298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:35:39.962306 15304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:35:39.962525 15304 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [term 1 LEADER]: Becoming Leader. State: Replica: 2b797c0520f445e285508efbc339532f, State: Running, Role: LEADER
I20260620 11:35:39.962666 15304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b797c0520f445e285508efbc339532f" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33281 } }
I20260620 11:35:39.963065 15304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b797c0520f445e285508efbc339532f. Latest consensus state: current_term: 1 leader_uuid: "2b797c0520f445e285508efbc339532f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b797c0520f445e285508efbc339532f" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33281 } } }
I20260620 11:35:39.963096 15303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b797c0520f445e285508efbc339532f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b797c0520f445e285508efbc339532f" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33281 } } }
I20260620 11:35:39.963215 15304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [sys.catalog]: This master's current role is: LEADER
I20260620 11:35:39.963235 15303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [sys.catalog]: This master's current role is: LEADER
I20260620 11:35:39.963881 15312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:35:39.964039 15298 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 11:35:39.964365 15312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 11:35:39 dist-test-slave-ctbw krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:35:39.973510 15301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:39.956754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51792 (local address 127.2.243.190:33281)
0620 11:35:39.956959 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:39.956969 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:39.957006 (+ 37us) server_negotiation.cc:407] Connection header received
0620 11:35:39.957536 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:39.957543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:39.957595 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:39.957785 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:39.958664 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:39.960065 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:39.961180 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:39.961788 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:39.961889 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:39.968439 (+ 6550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:39.968481 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:39.968482 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:39.968594 (+ 112us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:39.971467 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:39.971983 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:39.971986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:39.971986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:39.972092 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:39.972295 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:39.972296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:39.972296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:39.972720 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:39.972921 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:39.972993 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 11:35:39.973273 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":73,"threads_started":1}
I20260620 11:35:39.975412 15312 catalog_manager.cc:1347] Generated new cluster ID: 0379ac60899844de86a4dc2933ca638c
I20260620 11:35:39.975481 15312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:35:39.988729 15312 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:35:39.989744 15312 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:35:39.999198 15312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f: Generated new TSK 0
I20260620 11:35:39.999565 15312 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
I20260620 11:35:40.783260 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.243.148:49001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33281
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:35:40.876219 15343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:40.876466 15343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:40.876508 15343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:40.877779 15343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:40.877837 15343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:40.877882 15343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:35:40.879487 15343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.2.243.148:49001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:40.879745 15343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:40.879984 15343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:40.885040 15351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:40.885406 15348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:40.885470 15343 server_base.cc:1034] running on GCE node
W20260620 11:35:40.885681 15349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:40.885885 15343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:40.886173 15343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:40.887324 15343 hybrid_clock.cc:648] HybridClock initialized: now 1781955340887304 us; error 42 us; skew 500 ppm
Jun 20 11:35:40 dist-test-slave-ctbw krb5kdc[14786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955340, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:40.890390 15343 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:35:40.891008 15343 webserver.cc:460] Webserver started at http://127.2.243.129:33287/ using document root <none> and password file <none>
I20260620 11:35:40.891294 15343 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:40.891374 15343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:35:40.891531 15343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 11:35:40 dist-test-slave-ctbw krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955340, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for HTTP/127.2.243.148@KRBTEST.COM
I20260620 11:35:40.912299 15343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "6a6d8f9b0043411d96cf21aa23324d94"
format_stamp: "Formatted at 2026-06-20 11:35:40 on dist-test-slave-ctbw"
server_key: "080fdcb359cb3036c072247017a11976"
server_key_iv: "4cd5234c9cf8e10d6b8e37ed4eeac759"
server_key_version: "kuduclusterkey@0"
I20260620 11:35:40.912949 15343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "6a6d8f9b0043411d96cf21aa23324d94"
format_stamp: "Formatted at 2026-06-20 11:35:40 on dist-test-slave-ctbw"
server_key: "080fdcb359cb3036c072247017a11976"
server_key_iv: "4cd5234c9cf8e10d6b8e37ed4eeac759"
server_key_version: "kuduclusterkey@0"
I20260620 11:35:40.915182 15343 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 11:35:40.929561 15358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:40.929773 15343 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:35:40.929872 15343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "6a6d8f9b0043411d96cf21aa23324d94"
format_stamp: "Formatted at 2026-06-20 11:35:40 on dist-test-slave-ctbw"
server_key: "080fdcb359cb3036c072247017a11976"
server_key_iv: "4cd5234c9cf8e10d6b8e37ed4eeac759"
server_key_version: "kuduclusterkey@0"
I20260620 11:35:40.929976 15343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:40.957037 15343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:40.959357 15343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:40.959519 15343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:40.959748 15343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:35:40.960078 15343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:35:40.960134 15343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:40.960178 15343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:35:40.960206 15343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:40.967022 15343 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:38821
I20260620 11:35:40.967063 15471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:38821 every 8 connection(s)
I20260620 11:35:40.967448 15343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:35:40.969278 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 15343
I20260620 11:35:40.969363 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:35:40.969575 3022 external_mini_cluster.cc:1428] Setting key 2225f69973e11a1cea580e5a3d8b335c
Jun 20 11:35:40 dist-test-slave-ctbw krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955340, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.243.130@KRBTEST.COM; defaulting to no policy
I20260620 11:35:40.982223 15474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:40.968178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33281 (local address 127.2.243.129:43003)
0620 11:35:40.968468 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:40.968508 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:40.969093 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:40.970996 (+ 1903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:40.971002 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:40.971284 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:40.971616 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:40.971625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:40.972623 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:40.972625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:40.974617 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:40.974621 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:40.975348 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:40.975349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:40.975520 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:40.976108 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:40.976121 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:40.977877 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:40.980255 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:40.980262 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:40.980265 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:40.980611 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:40.980882 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:40.980883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:40.980883 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:40.981038 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:40.981418 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:40.981426 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:40.981613 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 11:35:40.981731 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 11:35:40.981911 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":108,"threads_started":1}
I20260620 11:35:40.982547 15475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:40.968263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:43003 (local address 127.2.243.190:33281)
0620 11:35:40.968398 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:40.968400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:40.968594 (+ 194us) server_negotiation.cc:407] Connection header received
0620 11:35:40.970076 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:40.970078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:40.970091 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:40.970190 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:40.971774 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:40.972481 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:40.974738 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:40.975154 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:40.975212 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:40.977988 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:40.977990 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:40.977991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:40.978002 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:40.980141 (+ 2139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:40.980717 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:40.980719 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:40.980720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:40.980797 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:40.981130 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:40.981131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:40.981132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:40.981300 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:40.981384 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:40.982347 (+ 963us) server_negotiation.cc:299] Negotiation successful
0620 11:35:40.982450 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260620 11:35:40.983165 15472 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33281
I20260620 11:35:40.983274 15472 heartbeater.cc:464] Registering TS with master...
I20260620 11:35:40.983460 15472 heartbeater.cc:511] Master 127.2.243.190:33281 requested a full tablet report, sending...
I20260620 11:35:40.984030 15262 ts_manager.cc:194] Registered new tserver with Master: 6a6d8f9b0043411d96cf21aa23324d94 (127.2.243.129:38821)
I20260620 11:35:40.984681 15262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:43003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
I20260620 11:35:41.006734 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.243.148:49001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33281
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:35:41.120982 15480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:41.121265 15480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:41.121305 15480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:41.123102 15480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:41.123165 15480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:41.123206 15480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:35:41.125391 15480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.2.243.148:49001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:41.125635 15480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:41.125950 15480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:41.132550 15488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:41.132668 15480 server_base.cc:1034] running on GCE node
W20260620 11:35:41.132915 15487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:41.133157 15490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:41.133384 15480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:41.133668 15480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:41.134845 15480 hybrid_clock.cc:648] HybridClock initialized: now 1781955341134812 us; error 41 us; skew 500 ppm
Jun 20 11:35:41 dist-test-slave-ctbw krb5kdc[14786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 11:35:41 dist-test-slave-ctbw krb5kdc[14786](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781955341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:41.138803 15480 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:35:41.139384 15480 webserver.cc:460] Webserver started at http://127.2.243.130:45075/ using document root <none> and password file <none>
I20260620 11:35:41.139669 15480 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:41.139750 15480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:35:41.139946 15480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 11:35:41 dist-test-slave-ctbw krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for HTTP/127.2.243.148@KRBTEST.COM
I20260620 11:35:41.159549 15480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "d13a1ca4f2684967bfccb9ccf2ca1725"
format_stamp: "Formatted at 2026-06-20 11:35:41 on dist-test-slave-ctbw"
server_key: "bafa6d897d6804e1a1726ed30bf94ab7"
server_key_iv: "568536a42846bc48d454ce522655570c"
server_key_version: "kuduclusterkey@0"
I20260620 11:35:41.160140 15480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "d13a1ca4f2684967bfccb9ccf2ca1725"
format_stamp: "Formatted at 2026-06-20 11:35:41 on dist-test-slave-ctbw"
server_key: "bafa6d897d6804e1a1726ed30bf94ab7"
server_key_iv: "568536a42846bc48d454ce522655570c"
server_key_version: "kuduclusterkey@0"
I20260620 11:35:41.167387 15480 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.000s
I20260620 11:35:41.187129 15497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:41.187395 15480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 11:35:41.187492 15480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "d13a1ca4f2684967bfccb9ccf2ca1725"
format_stamp: "Formatted at 2026-06-20 11:35:41 on dist-test-slave-ctbw"
server_key: "bafa6d897d6804e1a1726ed30bf94ab7"
server_key_iv: "568536a42846bc48d454ce522655570c"
server_key_version: "kuduclusterkey@0"
I20260620 11:35:41.187590 15480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:41.213014 15480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:41.215806 15480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:41.216027 15480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:41.216298 15480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:35:41.216720 15480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:35:41.216806 15480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:41.216868 15480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:35:41.216912 15480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:41.224094 15480 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:43685
I20260620 11:35:41.224184 15611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:43685 every 8 connection(s)
I20260620 11:35:41.224530 15480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:35:41.225623 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 15480
I20260620 11:35:41.225734 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:35:41.225981 3022 external_mini_cluster.cc:1428] Setting key 90d047a357422ecb8b5844f921d3609d
Jun 20 11:35:41 dist-test-slave-ctbw krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.131@KRBTEST.COM" created.
I20260620 11:35:41.245929 15475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:41.225321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:43851 (local address 127.2.243.190:33281)
0620 11:35:41.225369 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:41.225371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:41.225593 (+ 222us) server_negotiation.cc:407] Connection header received
0620 11:35:41.226262 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:41.226264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:41.226279 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:41.226333 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:41.227086 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:41.227855 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:41.229328 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:41.229526 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:41.229612 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:41.231977 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:41.231978 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:41.231979 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:41.231986 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:41.233806 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:41.234362 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:41.234363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:41.234363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:41.234425 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:41.234710 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:41.234711 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:41.234711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:41.237764 (+ 3053us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:41.237830 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:41.245655 (+ 7825us) server_negotiation.cc:299] Negotiation successful
0620 11:35:41.245809 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:35:41.246814 15614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:41.225279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33281 (local address 127.2.243.130:43851)
0620 11:35:41.225455 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:41.225498 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:41.226127 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:41.226416 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:41.226421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:41.226677 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:41.226988 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:41.226996 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:41.227962 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:41.227963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:41.229236 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:41.229239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:41.229615 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:41.229617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:41.229760 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:41.230336 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:41.230345 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:41.231868 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:41.233905 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:41.233912 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:41.233914 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:41.234242 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:41.234499 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:41.234500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:41.234500 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:41.234634 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:41.245047 (+ 10413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:41.245060 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:41.245377 (+ 317us) client_negotiation.cc:769] Sending connection context
0620 11:35:41.246331 (+ 954us) client_negotiation.cc:240] Negotiation successful
0620 11:35:41.246558 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":110,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
I20260620 11:35:41.256093 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.243.148:49001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33281
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:35:41.259472 15612 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33281
I20260620 11:35:41.259603 15612 heartbeater.cc:464] Registering TS with master...
I20260620 11:35:41.259835 15612 heartbeater.cc:511] Master 127.2.243.190:33281 requested a full tablet report, sending...
I20260620 11:35:41.260239 15262 ts_manager.cc:194] Registered new tserver with Master: d13a1ca4f2684967bfccb9ccf2ca1725 (127.2.243.130:43685)
I20260620 11:35:41.260797 15262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:43851
W20260620 11:35:41.391263 15617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:41.391628 15617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:41.391719 15617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:41.393513 15617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:41.393623 15617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:41.393707 15617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:35:41.396023 15617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.2.243.148:49001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:41.396497 15617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:41.396821 15617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:41.403913 15622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:41.404197 15623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:41.408229 15617 server_base.cc:1034] running on GCE node
W20260620 11:35:41.419103 15625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:41.419557 15617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:41.419934 15617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:41.423190 15617 hybrid_clock.cc:648] HybridClock initialized: now 1781955341423167 us; error 48 us; skew 500 ppm
Jun 20 11:35:41 dist-test-slave-ctbw krb5kdc[14786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:41.426837 15617 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:35:41.427423 15617 webserver.cc:460] Webserver started at http://127.2.243.131:33585/ using document root <none> and password file <none>
I20260620 11:35:41.427670 15617 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:41.427719 15617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:35:41.427842 15617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 11:35:41 dist-test-slave-ctbw krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for HTTP/127.2.243.148@KRBTEST.COM
I20260620 11:35:41.444283 15617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "02d6582f914a461bac61e8d776d8bd1f"
format_stamp: "Formatted at 2026-06-20 11:35:41 on dist-test-slave-ctbw"
server_key: "a38d5dbf5f08e23e982bbe4ff9731146"
server_key_iv: "1b27fa86f53a2367b061a3d9cc383f03"
server_key_version: "kuduclusterkey@0"
I20260620 11:35:41.444737 15617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "02d6582f914a461bac61e8d776d8bd1f"
format_stamp: "Formatted at 2026-06-20 11:35:41 on dist-test-slave-ctbw"
server_key: "a38d5dbf5f08e23e982bbe4ff9731146"
server_key_iv: "1b27fa86f53a2367b061a3d9cc383f03"
server_key_version: "kuduclusterkey@0"
I20260620 11:35:41.446651 15617 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 11:35:41.459769 15632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:41.460148 15617 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 11:35:41.460269 15617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "02d6582f914a461bac61e8d776d8bd1f"
format_stamp: "Formatted at 2026-06-20 11:35:41 on dist-test-slave-ctbw"
server_key: "a38d5dbf5f08e23e982bbe4ff9731146"
server_key_iv: "1b27fa86f53a2367b061a3d9cc383f03"
server_key_version: "kuduclusterkey@0"
I20260620 11:35:41.460376 15617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:41.486315 15617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:41.497296 15617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:41.497550 15617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:41.497905 15617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:35:41.498504 15617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:35:41.498612 15617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:41.498690 15617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:35:41.498749 15617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:41.505474 15617 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:43067
I20260620 11:35:41.505973 15617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:35:41.506800 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 15617
I20260620 11:35:41.506925 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:35:41.507136 3022 external_mini_cluster.cc:1428] Setting key 89a777957522c814b2019465d3593b6c
I20260620 11:35:41.509814 15745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:43067 every 8 connection(s)
Jun 20 11:35:41 dist-test-slave-ctbw krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:35:41.522045 15475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:41.510713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:51497 (local address 127.2.243.190:33281)
0620 11:35:41.510756 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:41.510758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:41.511119 (+ 361us) server_negotiation.cc:407] Connection header received
0620 11:35:41.511820 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:41.511821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:41.511833 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:41.511865 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:41.512722 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:41.513543 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:41.515083 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:41.515384 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:41.515427 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:41.518108 (+ 2681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:41.518110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:41.518110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:41.518118 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:41.520446 (+ 2328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:41.520924 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:41.520926 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:41.520926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:41.520982 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:41.521307 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:41.521308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:41.521308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:41.521447 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:41.521490 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:41.521854 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 11:35:41.521935 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:35:41.522943 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:41.510439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33281 (local address 127.2.243.131:51497)
0620 11:35:41.510980 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:41.511027 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:41.511691 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:41.512067 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:41.512071 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:41.512327 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:41.512608 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:41.512615 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:41.513661 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:41.513662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:41.514964 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:41.514966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:41.515525 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:41.515526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:41.515660 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:41.516316 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:41.516328 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:41.517979 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:41.520543 (+ 2564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:41.520550 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:41.520553 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:41.520839 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:41.521062 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:41.521063 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:41.521063 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:41.521192 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:41.521537 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:41.521543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:41.521706 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 11:35:41.522515 (+ 809us) client_negotiation.cc:240] Negotiation successful
0620 11:35:41.522681 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":84,"threads_started":1}
I20260620 11:35:41.523614 15746 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33281
I20260620 11:35:41.523710 15746 heartbeater.cc:464] Registering TS with master...
I20260620 11:35:41.523895 15746 heartbeater.cc:511] Master 127.2.243.190:33281 requested a full tablet report, sending...
I20260620 11:35:41.524318 15262 ts_manager.cc:194] Registered new tserver with Master: 02d6582f914a461bac61e8d776d8bd1f (127.2.243.131:43067)
I20260620 11:35:41.524791 15262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:51497
I20260620 11:35:41.531008 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:35:41.531862 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 15343
I20260620 11:35:41.539150 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 15480
I20260620 11:35:41.544651 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 15617
I20260620 11:35:41.549875 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 15230
I20260620 11:35:41.554919 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:33281
--webserver_interface=127.2.243.190
--webserver_port=39343
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:33281
--ranger_config_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.243.148:49001/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.243.148:49001/kms with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:35:41.637902 15749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:41.638154 15749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:41.638191 15749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:41.639643 15749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:35:41.639731 15749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:41.639757 15749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:41.639789 15749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:35:41.639813 15749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:35:41.641582 15749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.243.148:49001/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:33281
--ranger_config_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:33281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=39343
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:41.641852 15749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:41.642087 15749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:41.646921 15757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:41.646932 15754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:41.646974 15755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:41.647476 15749 server_base.cc:1034] running on GCE node
I20260620 11:35:41.647730 15749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:41.648023 15749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:41.649187 15749 hybrid_clock.cc:648] HybridClock initialized: now 1781955341649166 us; error 41 us; skew 500 ppm
Jun 20 11:35:41 dist-test-slave-ctbw krb5kdc[14786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:41.651824 15749 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:35:41.652303 15749 webserver.cc:460] Webserver started at http://127.2.243.190:39343/ using document root <none> and password file <none>
I20260620 11:35:41.652544 15749 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:41.652617 15749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 11:35:41 dist-test-slave-ctbw krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for HTTP/127.2.243.148@KRBTEST.COM
I20260620 11:35:41.690341 15749 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:35:41.691174 15765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:41.691381 15749 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:35:41.691499 15749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "2b797c0520f445e285508efbc339532f"
format_stamp: "Formatted at 2026-06-20 11:35:39 on dist-test-slave-ctbw"
server_key: "5c5f1cdd794570eeac139b85f8b92296"
server_key_iv: "9e4e64390669edb186d92c9f0b0ebd3b"
server_key_version: "kuduclusterkey@0"
I20260620 11:35:41.692001 15749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:41.711205 15749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:41.713358 15749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:41.713531 15749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:41.718585 15749 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:33281
I20260620 11:35:41.718612 15817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:33281 every 8 connection(s)
I20260620 11:35:41.719224 15749 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:35:41.720521 15818 sys_catalog.cc:263] Verifying existing consensus state
I20260620 11:35:41.720890 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 15749
I20260620 11:35:41.721200 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.243.148:49001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:38821
--local_ip_for_outbound_sockets=127.2.243.129
--tserver_master_addrs=127.2.243.190:33281
--webserver_port=33287
--webserver_interface=127.2.243.129
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.243.148:49001/kms with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:35:41.721406 15818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f: Bootstrap starting.
I20260620 11:35:41.729720 15818 log.cc:824] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f: Log is configured to *not* fsync() on all Append() calls
I20260620 11:35:41.731729 15818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:35:41.732065 15818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f: Bootstrap complete.
I20260620 11:35:41.734565 15818 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b797c0520f445e285508efbc339532f" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33281 } }
I20260620 11:35:41.734845 15818 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b797c0520f445e285508efbc339532f, State: Initialized, Role: FOLLOWER
I20260620 11:35:41.735040 15818 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b797c0520f445e285508efbc339532f" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33281 } }
I20260620 11:35:41.735114 15818 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:35:41.735159 15818 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:35:41.735208 15818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [term 1 FOLLOWER]: Advancing to term 2
I20260620 11:35:41.736347 15818 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b797c0520f445e285508efbc339532f" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33281 } }
I20260620 11:35:41.736492 15818 leader_election.cc:302] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b797c0520f445e285508efbc339532f; no voters:
I20260620 11:35:41.736701 15818 leader_election.cc:288] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 11:35:41.736811 15822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [term 2 FOLLOWER]: Leader election won for term 2
I20260620 11:35:41.736974 15822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [term 2 LEADER]: Becoming Leader. State: Replica: 2b797c0520f445e285508efbc339532f, State: Running, Role: LEADER
I20260620 11:35:41.736980 15818 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:35:41.737206 15822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b797c0520f445e285508efbc339532f" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33281 } }
I20260620 11:35:41.737543 15822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b797c0520f445e285508efbc339532f. Latest consensus state: current_term: 2 leader_uuid: "2b797c0520f445e285508efbc339532f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b797c0520f445e285508efbc339532f" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33281 } } }
I20260620 11:35:41.737707 15822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [sys.catalog]: This master's current role is: LEADER
I20260620 11:35:41.737959 15831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:35:41.738518 15831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:35:41.738705 15831 catalog_manager.cc:1259] Loaded cluster ID: 0379ac60899844de86a4dc2933ca638c
I20260620 11:35:41.738924 15831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:35:41.739161 15818 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 11:35:41.739590 15824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2b797c0520f445e285508efbc339532f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b797c0520f445e285508efbc339532f" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33281 } } }
I20260620 11:35:41.739665 15824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f [sys.catalog]: This master's current role is: LEADER
I20260620 11:35:41.743044 15831 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:35:41.743637 15831 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f: Loaded TSK: 0
I20260620 11:35:41.743963 15831 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 11:35:41.841099 15820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:41.841471 15820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:41.841558 15820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:41.843318 15820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:41.843441 15820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:41.843523 15820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:35:41.845693 15820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.243.148:49001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:38821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=33287
--tserver_master_addrs=127.2.243.190:33281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:41.846053 15820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:41.846323 15820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:41.852897 15856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:41.852903 15855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:41.853272 15858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:41.853521 15820 server_base.cc:1034] running on GCE node
I20260620 11:35:41.853734 15820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:41.853989 15820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:41.861436 15820 hybrid_clock.cc:648] HybridClock initialized: now 1781955341861376 us; error 61 us; skew 500 ppm
Jun 20 11:35:41 dist-test-slave-ctbw krb5kdc[14786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:41.865042 15820 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:35:41.865660 15820 webserver.cc:460] Webserver started at http://127.2.243.129:33287/ using document root <none> and password file <none>
I20260620 11:35:41.865962 15820 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:41.866107 15820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 11:35:41 dist-test-slave-ctbw krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for HTTP/127.2.243.148@KRBTEST.COM
I20260620 11:35:41.899264 15820 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 11:35:41.903273 15866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:41.903537 15820 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260620 11:35:41.903632 15820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "6a6d8f9b0043411d96cf21aa23324d94"
format_stamp: "Formatted at 2026-06-20 11:35:40 on dist-test-slave-ctbw"
server_key: "080fdcb359cb3036c072247017a11976"
server_key_iv: "4cd5234c9cf8e10d6b8e37ed4eeac759"
server_key_version: "kuduclusterkey@0"
I20260620 11:35:41.904116 15820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:41.923112 15820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:41.925844 15820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:41.926071 15820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:41.926370 15820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:35:41.926822 15820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:35:41.926951 15820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:41.927083 15820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:35:41.927173 15820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:41.933964 15820 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:38821
I20260620 11:35:41.934578 15820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:35:41.934919 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 15820
I20260620 11:35:41.935233 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.243.148:49001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:43685
--local_ip_for_outbound_sockets=127.2.243.130
--tserver_master_addrs=127.2.243.190:33281
--webserver_port=45075
--webserver_interface=127.2.243.130
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.243.148:49001/kms with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:35:41.939889 15979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:38821 every 8 connection(s)
Jun 20 11:35:41 dist-test-slave-ctbw krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955341, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:35:41.963709 15983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:41.940555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33281 (local address 127.2.243.129:44471)
0620 11:35:41.940877 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:41.940910 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:41.941411 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:41.941911 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:41.941915 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:41.942110 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:41.942296 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:41.942301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:41.948398 (+ 6097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:41.948399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:41.949401 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:41.949404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:41.950404 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:41.950406 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:41.950559 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:41.951252 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:41.951262 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:41.959181 (+ 7919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:41.961651 (+ 2470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:41.961659 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:41.961662 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:41.961979 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:41.962290 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:41.962291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:41.962291 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:41.962434 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:41.963003 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:41.963009 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:41.963169 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 11:35:41.963286 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 11:35:41.963470 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":101,"threads_started":1}
I20260620 11:35:41.963969 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:41.940593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:44471 (local address 127.2.243.190:33281)
0620 11:35:41.940785 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:41.940795 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:41.941016 (+ 221us) server_negotiation.cc:407] Connection header received
0620 11:35:41.941659 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:41.941664 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:41.941711 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:41.941855 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:41.942416 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:41.947539 (+ 5123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:41.949509 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:41.950154 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:41.950266 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:41.959356 (+ 9090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:41.959363 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:41.959363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:41.959380 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:41.961536 (+ 2156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:41.962085 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:41.962088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:41.962089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:41.962213 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:41.962509 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:41.962510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:41.962510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:41.962826 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:41.962981 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:41.963630 (+ 649us) server_negotiation.cc:299] Negotiation successful
0620 11:35:41.963770 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":73,"threads_started":1}
I20260620 11:35:41.964939 15980 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33281
I20260620 11:35:41.965036 15980 heartbeater.cc:464] Registering TS with master...
I20260620 11:35:41.965210 15980 heartbeater.cc:511] Master 127.2.243.190:33281 requested a full tablet report, sending...
W20260620 11:35:41.966017 15980 heartbeater.cc:644] Failed to heartbeat to 127.2.243.190:33281 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 11:35:41.966097 15980 heartbeater.cc:464] Registering TS with master...
I20260620 11:35:41.966164 15980 heartbeater.cc:511] Master 127.2.243.190:33281 requested a full tablet report, sending...
I20260620 11:35:41.966496 15980 heartbeater.cc:464] Registering TS with master...
I20260620 11:35:41.966595 15980 heartbeater.cc:511] Master 127.2.243.190:33281 requested a full tablet report, sending...
W20260620 11:35:41.966919 15980 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 11:35:42.045256 15982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:42.045614 15982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:42.045665 15982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:42.047369 15982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:42.047437 15982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:42.047514 15982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:35:42.049602 15982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.243.148:49001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:43685
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=45075
--tserver_master_addrs=127.2.243.190:33281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:42.049850 15982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:42.050146 15982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:42.056927 15990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:42.056953 15989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:42.056938 15992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:42.057268 15982 server_base.cc:1034] running on GCE node
I20260620 11:35:42.057574 15982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:42.057833 15982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:42.058936 15982 hybrid_clock.cc:648] HybridClock initialized: now 1781955342058915 us; error 36 us; skew 500 ppm
Jun 20 11:35:42 dist-test-slave-ctbw krb5kdc[14786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955342, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:42.063010 15982 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:35:42.063561 15982 webserver.cc:460] Webserver started at http://127.2.243.130:45075/ using document root <none> and password file <none>
I20260620 11:35:42.063872 15982 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:42.063951 15982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 11:35:42 dist-test-slave-ctbw krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955342, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for HTTP/127.2.243.148@KRBTEST.COM
I20260620 11:35:42.086072 15982 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260620 11:35:42.087520 16000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:42.095057 15982 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.000s
I20260620 11:35:42.095186 15982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "d13a1ca4f2684967bfccb9ccf2ca1725"
format_stamp: "Formatted at 2026-06-20 11:35:41 on dist-test-slave-ctbw"
server_key: "bafa6d897d6804e1a1726ed30bf94ab7"
server_key_iv: "568536a42846bc48d454ce522655570c"
server_key_version: "kuduclusterkey@0"
I20260620 11:35:42.095762 15982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:42.111387 15982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:42.113996 15982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:42.114190 15982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:42.114456 15982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:35:42.114918 15982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:35:42.114966 15982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:42.115023 15982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:35:42.115052 15982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:42.122251 15982 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:43685
I20260620 11:35:42.122351 16113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:43685 every 8 connection(s)
I20260620 11:35:42.122697 15982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:35:42.123011 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 15982
I20260620 11:35:42.123282 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.2.243.148:49001/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:43067
--local_ip_for_outbound_sockets=127.2.243.131
--tserver_master_addrs=127.2.243.190:33281
--webserver_port=33585
--webserver_interface=127.2.243.131
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.2.243.148:49001/kms with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 11:35:42 dist-test-slave-ctbw krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955342, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:35:42.133368 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:42.124845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:54943 (local address 127.2.243.190:33281)
0620 11:35:42.124904 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:42.124906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:42.124917 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:35:42.124947 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:42.124947 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:42.124962 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:42.125032 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:42.125847 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:42.126337 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:42.127519 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:42.127792 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:42.127830 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:42.130156 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:42.130157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:42.130157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:42.130165 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:42.131852 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:42.132255 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:42.132257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:42.132257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:42.132318 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:42.132579 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:42.132580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:42.132580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:42.132714 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:42.132789 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:42.133164 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 11:35:42.133252 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 11:35:42.133880 16116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:42.123496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33281 (local address 127.2.243.130:54943)
0620 11:35:42.123649 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:42.123692 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:42.124149 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:42.125171 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:42.125177 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:42.125414 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:42.125708 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:42.125716 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:42.126435 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:42.126436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:42.127417 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:42.127420 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:42.127921 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:42.127921 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:42.128019 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:42.128569 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:42.128578 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:42.130044 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:42.131947 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:42.131953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:42.131955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:42.132180 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:42.132386 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:42.132386 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:42.132387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:42.132509 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:42.132818 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:42.132824 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:42.133000 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 11:35:42.133534 (+ 534us) client_negotiation.cc:240] Negotiation successful
0620 11:35:42.133695 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260620 11:35:42.134402 16114 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33281
I20260620 11:35:42.134542 16114 heartbeater.cc:464] Registering TS with master...
I20260620 11:35:42.134771 16114 heartbeater.cc:511] Master 127.2.243.190:33281 requested a full tablet report, sending...
W20260620 11:35:42.135426 16114 heartbeater.cc:644] Failed to heartbeat to 127.2.243.190:33281 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 11:35:42.135637 16114 heartbeater.cc:464] Registering TS with master...
I20260620 11:35:42.135761 16114 heartbeater.cc:511] Master 127.2.243.190:33281 requested a full tablet report, sending...
I20260620 11:35:42.136111 16114 heartbeater.cc:464] Registering TS with master...
I20260620 11:35:42.136220 16114 heartbeater.cc:511] Master 127.2.243.190:33281 requested a full tablet report, sending...
W20260620 11:35:42.136549 16114 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 11:35:42.247421 16117 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:42.247802 16117 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:42.247911 16117 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:42.249454 16117 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:42.249565 16117 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:42.249655 16117 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:35:42.251966 16117 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.2.243.148:49001/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:43067
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=33585
--tserver_master_addrs=127.2.243.190:33281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16117
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:42.252228 16117 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:42.252513 16117 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:42.257911 16123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:42.257922 16126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:42.257963 16124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:42.258270 16117 server_base.cc:1034] running on GCE node
I20260620 11:35:42.258471 16117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:42.258723 16117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:42.259898 16117 hybrid_clock.cc:648] HybridClock initialized: now 1781955342259881 us; error 36 us; skew 500 ppm
Jun 20 11:35:42 dist-test-slave-ctbw krb5kdc[14786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955342, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:42.263011 16117 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:35:42.263608 16117 webserver.cc:460] Webserver started at http://127.2.243.131:33585/ using document root <none> and password file <none>
I20260620 11:35:42.263934 16117 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:42.264019 16117 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 11:35:42 dist-test-slave-ctbw krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955342, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for HTTP/127.2.243.148@KRBTEST.COM
I20260620 11:35:42.279002 16117 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:35:42.281985 16134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:42.282192 16117 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 11:35:42.282296 16117 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "02d6582f914a461bac61e8d776d8bd1f"
format_stamp: "Formatted at 2026-06-20 11:35:41 on dist-test-slave-ctbw"
server_key: "a38d5dbf5f08e23e982bbe4ff9731146"
server_key_iv: "1b27fa86f53a2367b061a3d9cc383f03"
server_key_version: "kuduclusterkey@0"
I20260620 11:35:42.282704 16117 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:42.291478 16117 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:42.293329 16117 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:42.293485 16117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:42.293727 16117 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:35:42.294055 16117 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:35:42.294088 16117 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:42.294111 16117 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:35:42.294157 16117 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:42.301716 16117 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:43067
I20260620 11:35:42.301802 16247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:43067 every 8 connection(s)
I20260620 11:35:42.302178 16117 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
Jun 20 11:35:42 dist-test-slave-ctbw krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955342, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:35:42.311127 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 16117
I20260620 11:35:42.313622 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:42.303008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:55085 (local address 127.2.243.190:33281)
0620 11:35:42.303056 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:42.303058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:42.303349 (+ 291us) server_negotiation.cc:407] Connection header received
0620 11:35:42.303978 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:42.303979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:42.303992 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:42.304065 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:42.304772 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:42.305539 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:42.307077 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:42.307397 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:42.307444 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:42.310083 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:42.310084 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:42.310085 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:42.310094 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:42.311949 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:42.312453 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:42.312454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:42.312455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:42.312523 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:42.312850 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:42.312851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:42.312852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:42.313012 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:42.313074 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:42.313438 (+ 364us) server_negotiation.cc:299] Negotiation successful
0620 11:35:42.313525 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:35:42.313825 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:42.302984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33281 (local address 127.2.243.131:55085)
0620 11:35:42.303177 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:42.303225 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:42.303846 (+ 621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:42.304135 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:42.304139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:42.304381 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:42.304655 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:42.304662 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:42.305653 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:42.305655 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:42.306970 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:42.306972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:42.307555 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:42.307556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:42.307693 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:42.308324 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:42.308335 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:42.309977 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:42.312052 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:42.312060 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:42.312062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:42.312333 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:42.312606 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:42.312607 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:42.312607 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:42.312758 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:42.313103 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:42.313109 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:42.313293 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 11:35:42.313407 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 11:35:42.313577 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":86,"threads_started":1}
I20260620 11:35:42.314313 16248 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33281
I20260620 11:35:42.314446 16248 heartbeater.cc:464] Registering TS with master...
I20260620 11:35:42.314661 16248 heartbeater.cc:511] Master 127.2.243.190:33281 requested a full tablet report, sending...
W20260620 11:35:42.315845 16248 heartbeater.cc:644] Failed to heartbeat to 127.2.243.190:33281 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 11:35:42.315958 16248 heartbeater.cc:464] Registering TS with master...
I20260620 11:35:42.316061 16248 heartbeater.cc:511] Master 127.2.243.190:33281 requested a full tablet report, sending...
I20260620 11:35:42.316442 16248 heartbeater.cc:464] Registering TS with master...
I20260620 11:35:42.316843 16248 heartbeater.cc:511] Master 127.2.243.190:33281 requested a full tablet report, sending...
W20260620 11:35:42.317412 16248 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 11:35:42.319289 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:42.311541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51808 (local address 127.2.243.190:33281)
0620 11:35:42.311674 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:42.311676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:42.311821 (+ 145us) server_negotiation.cc:407] Connection header received
0620 11:35:42.311892 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:42.311893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:42.311905 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:42.311993 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:42.312715 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:42.313235 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:42.314745 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:42.315030 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:42.315108 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:42.316544 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:42.316545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:42.316545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:42.316551 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:42.318065 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:42.318438 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:42.318439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:42.318439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:42.318493 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:42.318706 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:42.318707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:42.318708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:42.318832 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:42.319056 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:42.319095 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 11:35:42.319173 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
Jun 20 11:35:42 dist-test-slave-ctbw krb5kdc[14786](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781955342, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:42.967118 15980 heartbeater.cc:464] Registering TS with master...
I20260620 11:35:42.967284 15980 heartbeater.cc:511] Master 127.2.243.190:33281 requested a full tablet report, sending...
I20260620 11:35:42.967653 15782 ts_manager.cc:194] Registered new tserver with Master: 6a6d8f9b0043411d96cf21aa23324d94 (127.2.243.129:38821)
I20260620 11:35:42.968659 15782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:44471
I20260620 11:35:43.136757 16114 heartbeater.cc:464] Registering TS with master...
I20260620 11:35:43.136935 16114 heartbeater.cc:511] Master 127.2.243.190:33281 requested a full tablet report, sending...
I20260620 11:35:43.137264 15782 ts_manager.cc:194] Registered new tserver with Master: d13a1ca4f2684967bfccb9ccf2ca1725 (127.2.243.130:43685)
I20260620 11:35:43.137893 15782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:54943
Jun 20 11:35:43 dist-test-slave-ctbw krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955342, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for HTTP/127.2.243.148@KRBTEST.COM
I20260620 11:35:43.317667 16248 heartbeater.cc:464] Registering TS with master...
I20260620 11:35:43.317847 16248 heartbeater.cc:511] Master 127.2.243.190:33281 requested a full tablet report, sending...
I20260620 11:35:43.318182 15782 ts_manager.cc:194] Registered new tserver with Master: 02d6582f914a461bac61e8d776d8bd1f (127.2.243.131:43067)
I20260620 11:35:43.318732 15782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:55085
I20260620 11:35:43.327828 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:35:43.337390 16269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:43.328880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41396 (local address 127.2.243.190:33281)
0620 11:35:43.329101 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:43.329104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:43.329217 (+ 113us) server_negotiation.cc:407] Connection header received
0620 11:35:43.329317 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:43.329318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:43.329330 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:43.329373 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:43.330152 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.330987 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.332076 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.332402 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.332450 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:43.334069 (+ 1619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:43.334070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:43.334071 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:43.334078 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:43.335996 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:43.336396 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:43.336397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:43.336397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:43.336457 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:43.336672 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:43.336673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:43.336673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:43.336810 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:43.336893 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:43.337149 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 11:35:43.337283 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":63,"threads_started":1}
I20260620 11:35:43.340117 15782 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41396:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:35:43.341821 15782 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:35:43.350768 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:43.345742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39956 (local address 127.2.243.131:43067)
0620 11:35:43.345918 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:43.345924 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:43.346606 (+ 682us) server_negotiation.cc:407] Connection header received
0620 11:35:43.346716 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:43.346718 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:43.346733 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:43.346782 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:35:43.347062 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.348101 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.349495 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.350253 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.350305 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:43.350374 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:43.350637 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 11:35:43.350682 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":68,"threads_started":1}
I20260620 11:35:43.350806 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:43.345474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:43685 (local address 127.0.0.1:52056)
0620 11:35:43.346095 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:43.346136 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:43.346209 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:43.346462 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:43.346466 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:43.346468 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:35:43.346543 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:43.346545 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.347823 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.347824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:43.349370 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:43.349372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.350428 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.350429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:43.350511 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:43.350528 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 11:35:43.350592 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 11:35:43.350716 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"mutex_wait_us":5,"thread_start_us":93,"threads_started":1}
I20260620 11:35:43.351140 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:43.345558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52056 (local address 127.2.243.130:43685)
0620 11:35:43.346030 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:43.346033 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:43.346294 (+ 261us) server_negotiation.cc:407] Connection header received
0620 11:35:43.346331 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:43.346334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:43.346350 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:43.346437 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:35:43.346646 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.347720 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.349468 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.350260 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.350335 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:43.350395 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:43.351003 (+ 608us) server_negotiation.cc:299] Negotiation successful
0620 11:35:43.351050 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":55,"threads_started":1}
I20260620 11:35:43.351871 16048 tablet_service.cc:1467] Processing CreateTablet for tablet d53be1e659984bbb8aabf83c25a010a5 (DEFAULT_TABLE table=test-table [id=7983249fd31f4a7083c8c2b05b6dcb0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:35:43.352248 16048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d53be1e659984bbb8aabf83c25a010a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:35:43.355275 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:43.345807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:38821 (local address 127.0.0.1:52112)
0620 11:35:43.351256 (+ 5449us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:43.351267 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:43.351317 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:43.351616 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:43.351617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:43.351618 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:35:43.351660 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:43.351661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.352916 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.352916 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:43.353972 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:43.353975 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.355040 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.355041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:43.355104 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:43.355107 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:35:43.355156 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 11:35:43.355196 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5433,"thread_start_us":65,"threads_started":1}
I20260620 11:35:43.355547 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:43.345982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52112 (local address 127.2.243.129:38821)
0620 11:35:43.346188 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:43.346194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:43.351390 (+ 5196us) server_negotiation.cc:407] Connection header received
0620 11:35:43.351433 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:43.351435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:43.351452 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:43.351528 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:35:43.351740 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.352824 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.354072 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.354824 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.354944 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:43.355008 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:43.355418 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 11:35:43.355469 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":53,"threads_started":1}
I20260620 11:35:43.356220 15914 tablet_service.cc:1467] Processing CreateTablet for tablet d53be1e659984bbb8aabf83c25a010a5 (DEFAULT_TABLE table=test-table [id=7983249fd31f4a7083c8c2b05b6dcb0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:35:43.356556 15914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d53be1e659984bbb8aabf83c25a010a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:35:43.356767 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:43.345477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:43067 (local address 127.0.0.1:39956)
0620 11:35:43.346541 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:43.346551 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:43.346633 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:43.346903 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:43.346904 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:43.346906 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:35:43.346963 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:43.346964 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.348209 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.348209 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:43.349385 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:43.349389 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.350439 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.350440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:43.350531 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:43.350534 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:35:43.356632 (+ 6098us) client_negotiation.cc:240] Negotiation successful
0620 11:35:43.356689 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"thread_start_us":92,"threads_started":1}
I20260620 11:35:43.357240 16281 tablet_bootstrap.cc:492] T d53be1e659984bbb8aabf83c25a010a5 P d13a1ca4f2684967bfccb9ccf2ca1725: Bootstrap starting.
I20260620 11:35:43.357498 16182 tablet_service.cc:1467] Processing CreateTablet for tablet d53be1e659984bbb8aabf83c25a010a5 (DEFAULT_TABLE table=test-table [id=7983249fd31f4a7083c8c2b05b6dcb0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:35:43.357838 16182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d53be1e659984bbb8aabf83c25a010a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:35:43.357882 16281 tablet_bootstrap.cc:654] T d53be1e659984bbb8aabf83c25a010a5 P d13a1ca4f2684967bfccb9ccf2ca1725: Neither blocks nor log segments found. Creating new log.
I20260620 11:35:43.358178 16281 log.cc:824] T d53be1e659984bbb8aabf83c25a010a5 P d13a1ca4f2684967bfccb9ccf2ca1725: Log is configured to *not* fsync() on all Append() calls
I20260620 11:35:43.358999 16281 tablet_bootstrap.cc:492] T d53be1e659984bbb8aabf83c25a010a5 P d13a1ca4f2684967bfccb9ccf2ca1725: No bootstrap required, opened a new log
I20260620 11:35:43.359059 16281 ts_tablet_manager.cc:1397] T d53be1e659984bbb8aabf83c25a010a5 P d13a1ca4f2684967bfccb9ccf2ca1725: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:35:43.360594 16283 tablet_bootstrap.cc:492] T d53be1e659984bbb8aabf83c25a010a5 P 6a6d8f9b0043411d96cf21aa23324d94: Bootstrap starting.
I20260620 11:35:43.361325 16283 tablet_bootstrap.cc:654] T d53be1e659984bbb8aabf83c25a010a5 P 6a6d8f9b0043411d96cf21aa23324d94: Neither blocks nor log segments found. Creating new log.
I20260620 11:35:43.361487 16281 raft_consensus.cc:348] T d53be1e659984bbb8aabf83c25a010a5 P d13a1ca4f2684967bfccb9ccf2ca1725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13a1ca4f2684967bfccb9ccf2ca1725" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 43685 } } peers { permanent_uuid: "6a6d8f9b0043411d96cf21aa23324d94" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38821 } } peers { permanent_uuid: "02d6582f914a461bac61e8d776d8bd1f" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43067 } }
I20260620 11:35:43.361649 16281 raft_consensus.cc:374] T d53be1e659984bbb8aabf83c25a010a5 P d13a1ca4f2684967bfccb9ccf2ca1725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:35:43.361680 16281 raft_consensus.cc:729] T d53be1e659984bbb8aabf83c25a010a5 P d13a1ca4f2684967bfccb9ccf2ca1725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13a1ca4f2684967bfccb9ccf2ca1725, State: Initialized, Role: FOLLOWER
I20260620 11:35:43.361742 16283 log.cc:824] T d53be1e659984bbb8aabf83c25a010a5 P 6a6d8f9b0043411d96cf21aa23324d94: Log is configured to *not* fsync() on all Append() calls
I20260620 11:35:43.361788 16281 consensus_queue.cc:260] T d53be1e659984bbb8aabf83c25a010a5 P d13a1ca4f2684967bfccb9ccf2ca1725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13a1ca4f2684967bfccb9ccf2ca1725" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 43685 } } peers { permanent_uuid: "6a6d8f9b0043411d96cf21aa23324d94" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38821 } } peers { permanent_uuid: "02d6582f914a461bac61e8d776d8bd1f" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43067 } }
I20260620 11:35:43.362030 16281 ts_tablet_manager.cc:1428] T d53be1e659984bbb8aabf83c25a010a5 P d13a1ca4f2684967bfccb9ccf2ca1725: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 11:35:43.362179 16114 heartbeater.cc:503] Master 127.2.243.190:33281 was elected leader, sending a full tablet report...
I20260620 11:35:43.362999 16283 tablet_bootstrap.cc:492] T d53be1e659984bbb8aabf83c25a010a5 P 6a6d8f9b0043411d96cf21aa23324d94: No bootstrap required, opened a new log
I20260620 11:35:43.363113 16283 ts_tablet_manager.cc:1397] T d53be1e659984bbb8aabf83c25a010a5 P 6a6d8f9b0043411d96cf21aa23324d94: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 11:35:43.365537 16283 raft_consensus.cc:348] T d53be1e659984bbb8aabf83c25a010a5 P 6a6d8f9b0043411d96cf21aa23324d94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13a1ca4f2684967bfccb9ccf2ca1725" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 43685 } } peers { permanent_uuid: "6a6d8f9b0043411d96cf21aa23324d94" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38821 } } peers { permanent_uuid: "02d6582f914a461bac61e8d776d8bd1f" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43067 } }
I20260620 11:35:43.365700 16283 raft_consensus.cc:374] T d53be1e659984bbb8aabf83c25a010a5 P 6a6d8f9b0043411d96cf21aa23324d94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:35:43.365731 16283 raft_consensus.cc:729] T d53be1e659984bbb8aabf83c25a010a5 P 6a6d8f9b0043411d96cf21aa23324d94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6d8f9b0043411d96cf21aa23324d94, State: Initialized, Role: FOLLOWER
I20260620 11:35:43.365875 16283 consensus_queue.cc:260] T d53be1e659984bbb8aabf83c25a010a5 P 6a6d8f9b0043411d96cf21aa23324d94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13a1ca4f2684967bfccb9ccf2ca1725" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 43685 } } peers { permanent_uuid: "6a6d8f9b0043411d96cf21aa23324d94" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38821 } } peers { permanent_uuid: "02d6582f914a461bac61e8d776d8bd1f" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43067 } }
I20260620 11:35:43.366164 16283 ts_tablet_manager.cc:1428] T d53be1e659984bbb8aabf83c25a010a5 P 6a6d8f9b0043411d96cf21aa23324d94: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:35:43.366196 15980 heartbeater.cc:503] Master 127.2.243.190:33281 was elected leader, sending a full tablet report...
I20260620 11:35:43.366643 16284 tablet_bootstrap.cc:492] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f: Bootstrap starting.
I20260620 11:35:43.367378 16284 tablet_bootstrap.cc:654] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f: Neither blocks nor log segments found. Creating new log.
I20260620 11:35:43.367687 16284 log.cc:824] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f: Log is configured to *not* fsync() on all Append() calls
I20260620 11:35:43.368556 16284 tablet_bootstrap.cc:492] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f: No bootstrap required, opened a new log
I20260620 11:35:43.368634 16284 ts_tablet_manager.cc:1397] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:35:43.371160 16284 raft_consensus.cc:348] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13a1ca4f2684967bfccb9ccf2ca1725" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 43685 } } peers { permanent_uuid: "6a6d8f9b0043411d96cf21aa23324d94" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38821 } } peers { permanent_uuid: "02d6582f914a461bac61e8d776d8bd1f" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43067 } }
I20260620 11:35:43.371333 16284 raft_consensus.cc:374] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:35:43.371376 16284 raft_consensus.cc:729] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d6582f914a461bac61e8d776d8bd1f, State: Initialized, Role: FOLLOWER
I20260620 11:35:43.371548 16284 consensus_queue.cc:260] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13a1ca4f2684967bfccb9ccf2ca1725" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 43685 } } peers { permanent_uuid: "6a6d8f9b0043411d96cf21aa23324d94" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38821 } } peers { permanent_uuid: "02d6582f914a461bac61e8d776d8bd1f" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43067 } }
I20260620 11:35:43.371815 16284 ts_tablet_manager.cc:1428] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:35:43.371837 16248 heartbeater.cc:503] Master 127.2.243.190:33281 was elected leader, sending a full tablet report...
W20260620 11:35:43.374015 16115 tablet.cc:2369] T d53be1e659984bbb8aabf83c25a010a5 P d13a1ca4f2684967bfccb9ccf2ca1725: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 11:35:43.441385 15981 tablet.cc:2369] T d53be1e659984bbb8aabf83c25a010a5 P 6a6d8f9b0043411d96cf21aa23324d94: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 11:35:43 dist-test-slave-ctbw krb5kdc[14786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955342, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for HTTP/127.2.243.148@KRBTEST.COM
I20260620 11:35:43.485131 16289 raft_consensus.cc:482] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:35:43.485303 16289 raft_consensus.cc:504] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13a1ca4f2684967bfccb9ccf2ca1725" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 43685 } } peers { permanent_uuid: "6a6d8f9b0043411d96cf21aa23324d94" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38821 } } peers { permanent_uuid: "02d6582f914a461bac61e8d776d8bd1f" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43067 } }
I20260620 11:35:43.485715 16289 leader_election.cc:288] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d13a1ca4f2684967bfccb9ccf2ca1725 (127.2.243.130:43685), 6a6d8f9b0043411d96cf21aa23324d94 (127.2.243.129:38821)
I20260620 11:35:43.492700 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:43.486013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:36653 (local address 127.2.243.130:43685)
0620 11:35:43.486061 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:43.486064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:43.488424 (+ 2360us) server_negotiation.cc:407] Connection header received
0620 11:35:43.489161 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:43.489162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:43.489173 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:43.489202 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:35:43.489422 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.490134 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.491686 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.492237 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.492290 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:43.492317 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:43.492567 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 11:35:43.492626 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:35:43.492966 16292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:43.485942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:43685 (local address 127.2.243.131:36653)
0620 11:35:43.488184 (+ 2242us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:43.488355 (+ 171us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:43.489045 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:43.489274 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:43.489275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:43.489276 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:35:43.489342 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:43.489344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.490236 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.490237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:43.491565 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:43.491569 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.492401 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.492402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:43.492476 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:43.492481 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:35:43.492804 (+ 323us) client_negotiation.cc:240] Negotiation successful
0620 11:35:43.492861 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2027,"thread_start_us":113,"threads_started":1}
I20260620 11:35:43.493366 16066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53be1e659984bbb8aabf83c25a010a5" candidate_uuid: "02d6582f914a461bac61e8d776d8bd1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13a1ca4f2684967bfccb9ccf2ca1725" is_pre_election: true
I20260620 11:35:43.493534 16066 raft_consensus.cc:2454] T d53be1e659984bbb8aabf83c25a010a5 P d13a1ca4f2684967bfccb9ccf2ca1725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d6582f914a461bac61e8d776d8bd1f in term 0.
I20260620 11:35:43.493777 16136 leader_election.cc:302] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d6582f914a461bac61e8d776d8bd1f, d13a1ca4f2684967bfccb9ccf2ca1725; no voters:
I20260620 11:35:43.493945 16289 raft_consensus.cc:2781] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:35:43.493996 16289 raft_consensus.cc:482] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:35:43.494024 16289 raft_consensus.cc:3037] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:35:43.494969 16289 raft_consensus.cc:504] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13a1ca4f2684967bfccb9ccf2ca1725" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 43685 } } peers { permanent_uuid: "6a6d8f9b0043411d96cf21aa23324d94" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38821 } } peers { permanent_uuid: "02d6582f914a461bac61e8d776d8bd1f" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43067 } }
I20260620 11:35:43.495085 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:43.486328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:60797 (local address 127.2.243.129:38821)
0620 11:35:43.486368 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:43.486370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:43.491573 (+ 5203us) server_negotiation.cc:407] Connection header received
0620 11:35:43.491677 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:43.491678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:43.491688 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:43.491725 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:35:43.491968 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.492729 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.494140 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.494681 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.494772 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:43.494782 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:43.494978 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 11:35:43.495023 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:35:43.495124 16289 leader_election.cc:288] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f [CANDIDATE]: Term 1 election: Requested vote from peers d13a1ca4f2684967bfccb9ccf2ca1725 (127.2.243.130:43685), 6a6d8f9b0043411d96cf21aa23324d94 (127.2.243.129:38821)
I20260620 11:35:43.495100 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:43.486177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:38821 (local address 127.2.243.131:60797)
0620 11:35:43.491494 (+ 5317us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:43.491507 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:43.491604 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:43.491812 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:43.491813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:43.491814 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:35:43.491882 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:43.491883 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.492833 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.492834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:43.494008 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:43.494011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.494785 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.494787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:43.494897 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:43.494901 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:35:43.494970 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 11:35:43.495012 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5293,"thread_start_us":58,"threads_started":1}
I20260620 11:35:43.495507 16066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53be1e659984bbb8aabf83c25a010a5" candidate_uuid: "02d6582f914a461bac61e8d776d8bd1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d13a1ca4f2684967bfccb9ccf2ca1725"
I20260620 11:35:43.495596 16066 raft_consensus.cc:3037] T d53be1e659984bbb8aabf83c25a010a5 P d13a1ca4f2684967bfccb9ccf2ca1725 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:35:43.495641 15934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53be1e659984bbb8aabf83c25a010a5" candidate_uuid: "02d6582f914a461bac61e8d776d8bd1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6d8f9b0043411d96cf21aa23324d94" is_pre_election: true
I20260620 11:35:43.495807 15934 raft_consensus.cc:2454] T d53be1e659984bbb8aabf83c25a010a5 P 6a6d8f9b0043411d96cf21aa23324d94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02d6582f914a461bac61e8d776d8bd1f in term 0.
I20260620 11:35:43.495923 15933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d53be1e659984bbb8aabf83c25a010a5" candidate_uuid: "02d6582f914a461bac61e8d776d8bd1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a6d8f9b0043411d96cf21aa23324d94"
I20260620 11:35:43.495965 15933 raft_consensus.cc:3037] T d53be1e659984bbb8aabf83c25a010a5 P 6a6d8f9b0043411d96cf21aa23324d94 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:35:43.496515 16066 raft_consensus.cc:2454] T d53be1e659984bbb8aabf83c25a010a5 P d13a1ca4f2684967bfccb9ccf2ca1725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d6582f914a461bac61e8d776d8bd1f in term 1.
I20260620 11:35:43.496722 16136 leader_election.cc:302] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d6582f914a461bac61e8d776d8bd1f, d13a1ca4f2684967bfccb9ccf2ca1725; no voters:
I20260620 11:35:43.496816 15933 raft_consensus.cc:2454] T d53be1e659984bbb8aabf83c25a010a5 P 6a6d8f9b0043411d96cf21aa23324d94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02d6582f914a461bac61e8d776d8bd1f in term 1.
I20260620 11:35:43.496933 16289 raft_consensus.cc:2781] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:35:43.497092 16289 raft_consensus.cc:686] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f [term 1 LEADER]: Becoming Leader. State: Replica: 02d6582f914a461bac61e8d776d8bd1f, State: Running, Role: LEADER
I20260620 11:35:43.497182 16289 consensus_queue.cc:237] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13a1ca4f2684967bfccb9ccf2ca1725" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 43685 } } peers { permanent_uuid: "6a6d8f9b0043411d96cf21aa23324d94" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38821 } } peers { permanent_uuid: "02d6582f914a461bac61e8d776d8bd1f" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43067 } }
I20260620 11:35:43.498034 15782 catalog_manager.cc:5495] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 02d6582f914a461bac61e8d776d8bd1f (127.2.243.131). New cstate: current_term: 1 leader_uuid: "02d6582f914a461bac61e8d776d8bd1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d13a1ca4f2684967bfccb9ccf2ca1725" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 43685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a6d8f9b0043411d96cf21aa23324d94" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02d6582f914a461bac61e8d776d8bd1f" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 43067 } health_report { overall_health: HEALTHY } } }
I20260620 11:35:43.530380 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:43.527245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39962 (local address 127.2.243.131:43067)
0620 11:35:43.527314 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:43.527317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:43.527328 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:35:43.527373 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:43.527374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:43.527396 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:43.527486 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:35:43.527669 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.528335 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.529288 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:43.529572 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:43.529654 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:43.529819 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:35:43.529957 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:35:43.530035 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:43.530275 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 11:35:43.530299 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 11:35:43.534152 16066 raft_consensus.cc:1261] T d53be1e659984bbb8aabf83c25a010a5 P d13a1ca4f2684967bfccb9ccf2ca1725 [term 1 FOLLOWER]: Refusing update from remote peer 02d6582f914a461bac61e8d776d8bd1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:35:43.534462 15933 raft_consensus.cc:1261] T d53be1e659984bbb8aabf83c25a010a5 P 6a6d8f9b0043411d96cf21aa23324d94 [term 1 FOLLOWER]: Refusing update from remote peer 02d6582f914a461bac61e8d776d8bd1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:35:43.534608 16289 consensus_queue.cc:1035] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d13a1ca4f2684967bfccb9ccf2ca1725" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 43685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:35:43.534790 16289 consensus_queue.cc:1035] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a6d8f9b0043411d96cf21aa23324d94" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:35:43.539295 16298 mvcc.cc:205] Tried to move back new op lower bound from 7298889087113216000 to 7298889086965506048. Current Snapshot: MvccSnapshot[applied={T|T < 7298889087113216000}]
I20260620 11:35:43.539652 16297 mvcc.cc:205] Tried to move back new op lower bound from 7298889087113216000 to 7298889086965506048. Current Snapshot: MvccSnapshot[applied={T|T < 7298889087113216000}]
I20260620 11:35:43.542620 15782 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:35:43.542722 15782 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41396:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:35:43.543450 15782 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b797c0520f445e285508efbc339532f: Sending DeleteTablet for 3 replicas of tablet d53be1e659984bbb8aabf83c25a010a5
I20260620 11:35:43.543828 16048 tablet_service.cc:1514] Processing DeleteTablet for tablet d53be1e659984bbb8aabf83c25a010a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:35:43 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:52056
I20260620 11:35:43.544729 16300 mvcc.cc:205] Tried to move back new op lower bound from 7298889087113216000 to 7298889086965506048. Current Snapshot: MvccSnapshot[applied={T|T < 7298889087113216000}]
I20260620 11:35:43.545526 15914 tablet_service.cc:1514] Processing DeleteTablet for tablet d53be1e659984bbb8aabf83c25a010a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:35:43 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:52112
I20260620 11:35:43.545994 16305 tablet_replica.cc:330] T d53be1e659984bbb8aabf83c25a010a5 P d13a1ca4f2684967bfccb9ccf2ca1725: stopping tablet replica
I20260620 11:35:43.545998 16182 tablet_service.cc:1514] Processing DeleteTablet for tablet d53be1e659984bbb8aabf83c25a010a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:35:43 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:39956
I20260620 11:35:43.546092 16305 raft_consensus.cc:2229] T d53be1e659984bbb8aabf83c25a010a5 P d13a1ca4f2684967bfccb9ccf2ca1725 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:35:43.546185 16305 raft_consensus.cc:2258] T d53be1e659984bbb8aabf83c25a010a5 P d13a1ca4f2684967bfccb9ccf2ca1725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:35:43.546278 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 15820
I20260620 11:35:43.551230 16305 ts_tablet_manager.cc:1905] T d53be1e659984bbb8aabf83c25a010a5 P d13a1ca4f2684967bfccb9ccf2ca1725: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:35:43.552934 16305 ts_tablet_manager.cc:1918] T d53be1e659984bbb8aabf83c25a010a5 P d13a1ca4f2684967bfccb9ccf2ca1725: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:35:43.553095 16305 log.cc:1196] T d53be1e659984bbb8aabf83c25a010a5 P d13a1ca4f2684967bfccb9ccf2ca1725: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-1/wal/wals/d53be1e659984bbb8aabf83c25a010a5
I20260620 11:35:43.553650 16305 ts_tablet_manager.cc:1939] T d53be1e659984bbb8aabf83c25a010a5 P d13a1ca4f2684967bfccb9ccf2ca1725: Deleting consensus metadata
I20260620 11:35:43.554162 15767 catalog_manager.cc:4841] TS d13a1ca4f2684967bfccb9ccf2ca1725 (127.2.243.130:43685): tablet d53be1e659984bbb8aabf83c25a010a5 (table test-table [id=7983249fd31f4a7083c8c2b05b6dcb0e]) successfully deleted
I20260620 11:35:43.584839 16308 tablet_replica.cc:330] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f: stopping tablet replica
I20260620 11:35:43.585144 16308 raft_consensus.cc:2229] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:35:43.585371 16308 raft_consensus.cc:2258] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:35:43.586915 16308 ts_tablet_manager.cc:1905] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:35:43.599108 16308 ts_tablet_manager.cc:1918] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:35:43.599325 16308 log.cc:1196] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/d53be1e659984bbb8aabf83c25a010a5
I20260620 11:35:43.599786 16308 ts_tablet_manager.cc:1939] T d53be1e659984bbb8aabf83c25a010a5 P 02d6582f914a461bac61e8d776d8bd1f: Deleting consensus metadata
I20260620 11:35:43.600396 15768 catalog_manager.cc:4841] TS 02d6582f914a461bac61e8d776d8bd1f (127.2.243.131:43067): tablet d53be1e659984bbb8aabf83c25a010a5 (table test-table [id=7983249fd31f4a7083c8c2b05b6dcb0e]) successfully deleted
W20260620 11:35:43.600620 15767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.243.129:38821 (error 108)
I20260620 11:35:43.601387 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 15982
I20260620 11:35:43.608170 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 16117
I20260620 11:35:43.615229 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:43.615054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:38821 (local address 127.0.0.1:52128)
0620 11:35:43.615117 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:43.615174 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.129:38821: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 11:35:43.615556 15767 catalog_manager.cc:4584] TS 6a6d8f9b0043411d96cf21aa23324d94 (127.2.243.129:38821): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d53be1e659984bbb8aabf83c25a010a5: Network error: Client connection negotiation failed: client connection to 127.2.243.129:38821: connect: Connection refused (error 111)
I20260620 11:35:43.619050 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 15749
I20260620 11:35:43.626062 3022 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 11:35:43.970829 3022 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 11:35:43.971006 3022 mini_ranger.cc:66] Stopping Ranger...
I20260620 11:35:44.315302 3022 mini_ranger.cc:68] Stopped Ranger
2026-06-20 11:35:44.315 UTC [14816] LOG: received smart shutdown request
2026-06-20 11:35:44.318 UTC [14816] LOG: background worker "logical replication launcher" (PID 14825) exited with exit code 1
2026-06-20 11:35:44.319 UTC [14820] LOG: shutting down
2026-06-20 11:35:44.470 UTC [14816] LOG: database system is shut down
2026-06-20T11:35:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48089 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:35:44 dist-test-slave-ctbw krb5kdc[16318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:35:44 dist-test-slave-ctbw krb5kdc[16318](info): set up 2 sockets
Jun 20 11:35:44 dist-test-slave-ctbw krb5kdc[16318](info): commencing operation
krb5kdc: starting...
W20260620 11:35:45.627452 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:35:45 dist-test-slave-ctbw krb5kdc[16318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:35:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:35:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:35:45.753472 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:36401
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:44649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:36401,127.2.243.189:39643,127.2.243.188:39461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:35:45.837885 16331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:45.838173 16331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:45.838212 16331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:45.839495 16331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:35:45.839567 16331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:45.839591 16331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:45.839623 16331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:35:45.839650 16331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:35:45.841229 16331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:44649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:36401,127.2.243.189:39643,127.2.243.188:39461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:36401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:45.841534 16331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:45.841778 16331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:45.846807 16337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:45.846791 16339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:45.846948 16336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:45.847158 16331 server_base.cc:1034] running on GCE node
I20260620 11:35:45.847467 16331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:45.847865 16331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:45.849051 16331 hybrid_clock.cc:648] HybridClock initialized: now 1781955345849020 us; error 62 us; skew 500 ppm
Jun 20 11:35:45 dist-test-slave-ctbw krb5kdc[16318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955345, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:45.854914 16331 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:35:45.855480 16331 webserver.cc:460] Webserver started at http://127.2.243.190:45813/ using document root <none> and password file <none>
I20260620 11:35:45.855744 16331 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:45.855818 16331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:35:45.855973 16331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:35:45.857131 16331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "d3f604dbffb44b0ca15d06c0c0696458"
format_stamp: "Formatted at 2026-06-20 11:35:45 on dist-test-slave-ctbw"
server_key: "6dc76c6d62a664f5a6f0288cf0babfcd"
server_key_iv: "e6644f0db9e25c130c494b56daae8bb4"
server_key_version: "encryptionkey@0"
I20260620 11:35:45.857512 16331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "d3f604dbffb44b0ca15d06c0c0696458"
format_stamp: "Formatted at 2026-06-20 11:35:45 on dist-test-slave-ctbw"
server_key: "6dc76c6d62a664f5a6f0288cf0babfcd"
server_key_iv: "e6644f0db9e25c130c494b56daae8bb4"
server_key_version: "encryptionkey@0"
I20260620 11:35:45.859035 16331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 11:35:45.859890 16346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:45.860101 16331 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:35:45.860214 16331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "d3f604dbffb44b0ca15d06c0c0696458"
format_stamp: "Formatted at 2026-06-20 11:35:45 on dist-test-slave-ctbw"
server_key: "6dc76c6d62a664f5a6f0288cf0babfcd"
server_key_iv: "e6644f0db9e25c130c494b56daae8bb4"
server_key_version: "encryptionkey@0"
I20260620 11:35:45.860296 16331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:45.876144 16331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:45.878305 16331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:45.878512 16331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:45.882756 16331 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:36401
I20260620 11:35:45.882787 16398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:36401 every 8 connection(s)
I20260620 11:35:45.883219 16331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:35:45.884145 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:35:45.885746 16399 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36401 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:35:45.888782 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 16331
I20260620 11:35:45.888878 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:35:45.889103 3022 external_mini_cluster.cc:1428] Setting key 47ed4647488c4edf8cda02a6da9095e7
Jun 20 11:35:45 dist-test-slave-ctbw krb5kdc[16318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955345, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.243.189@KRBTEST.COM; defaulting to no policy
I20260620 11:35:45.898785 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:45.886462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:36401 (local address 127.0.0.1:33918)
0620 11:35:45.886785 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:45.886828 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:45.887363 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:45.887634 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:45.887636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:45.887897 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:45.888082 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:45.888086 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:45.889283 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:45.889285 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:45.890481 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:45.890485 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:45.891253 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:45.891254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:45.891351 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:45.892047 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:45.892059 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:45.894258 (+ 2199us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:45.896728 (+ 2470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:45.896737 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:45.896758 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:45.897054 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:45.897374 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:45.897375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:45.897375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:45.897512 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:45.898051 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:45.898059 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:45.898146 (+ 87us) client_negotiation.cc:769] Sending connection context
0620 11:35:45.898261 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 11:35:45.898535 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":112,"thread_start_us":242,"threads_started":1}
I20260620 11:35:45.898783 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:45.886503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33918 (local address 127.2.243.190:36401)
0620 11:35:45.886843 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:45.886847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:45.886982 (+ 135us) server_negotiation.cc:407] Connection header received
0620 11:35:45.887491 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:45.887493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:45.887510 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:45.887606 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:45.888165 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:45.889148 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:45.890584 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:45.891150 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:45.891289 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:45.894368 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:45.894377 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:45.894378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:45.894386 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:45.896604 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:45.897164 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:45.897168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:45.897168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:45.897284 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:45.897594 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:45.897595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:45.897596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:45.897947 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:45.898050 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:45.898311 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 11:35:45.898463 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":281,"threads_started":1}
I20260620 11:35:45.900213 16399 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:35:45.900651 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:45.900523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:39643 (local address 127.0.0.1:37798)
0620 11:35:45.900577 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:45.900625 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.189:39643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 11:35:45.900805 16350 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.243.189:39643: connect: Connection refused (error 111)
I20260620 11:35:45.901289 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:45.901158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:39643 (local address 127.0.0.1:37812)
0620 11:35:45.901221 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:45.901265 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.189:39643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.189@KRBTEST.COM" created.
W20260620 11:35:45.901415 16399 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.189:39643: Network error: Client connection negotiation failed: client connection to 127.2.243.189:39643: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.189.keytab.
Entry for principal kudu/127.2.243.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.189.keytab.
I20260620 11:35:45.917363 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.189:39643
--webserver_interface=127.2.243.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.189.keytab
--principal=kudu/127.2.243.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:44649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:36401,127.2.243.189:39643,127.2.243.188:39461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:35:45.950645 16399 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } attempt: 1
I20260620 11:35:45.951236 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:45.951051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:39643 (local address 127.0.0.1:37816)
0620 11:35:45.951133 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:45.951185 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.189:39643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 11:35:45.951651 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:45.951554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:39643 (local address 127.0.0.1:37830)
0620 11:35:45.951596 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:45.951620 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.189:39643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260620 11:35:45.951778 16399 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.189:39643: Network error: Client connection negotiation failed: client connection to 127.2.243.189:39643: connect: Connection refused (error 111)
W20260620 11:35:46.002771 16406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:46.003029 16406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:46.003055 16406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:46.004370 16406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:35:46.004429 16406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:46.004443 16406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:46.004457 16406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:35:46.004469 16406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:35:46.006181 16406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:44649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:36401,127.2.243.189:39643,127.2.243.188:39461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.189.keytab
--principal=kudu/127.2.243.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.189:39643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.243.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:46.006477 16406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:46.006731 16406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:46.011905 16411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:46.011912 16414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:46.011940 16412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:46.012378 16406 server_base.cc:1034] running on GCE node
I20260620 11:35:46.012701 16406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:46.013118 16406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:46.014298 16406 hybrid_clock.cc:648] HybridClock initialized: now 1781955346014270 us; error 47 us; skew 500 ppm
Jun 20 11:35:46 dist-test-slave-ctbw krb5kdc[16318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955346, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:46.017643 16406 init.cc:377] Logged in from keytab as kudu/127.2.243.189@KRBTEST.COM (short username kudu)
I20260620 11:35:46.018154 16406 webserver.cc:460] Webserver started at http://127.2.243.189:37325/ using document root <none> and password file <none>
I20260620 11:35:46.018405 16406 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:46.018457 16406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:35:46.018564 16406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:35:46.019984 16399 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } attempt: 2
I20260620 11:35:46.020071 16406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-1/data/instance:
uuid: "92184a70a3304f82aa3c183ce425fe61"
format_stamp: "Formatted at 2026-06-20 11:35:46 on dist-test-slave-ctbw"
server_key: "5e06985f16e1fbc9d98df48cd35dc4cf"
server_key_iv: "d03551b73f53123ca3153d112df01eed"
server_key_version: "encryptionkey@0"
I20260620 11:35:46.020512 16406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-1/wal/instance:
uuid: "92184a70a3304f82aa3c183ce425fe61"
format_stamp: "Formatted at 2026-06-20 11:35:46 on dist-test-slave-ctbw"
server_key: "5e06985f16e1fbc9d98df48cd35dc4cf"
server_key_iv: "d03551b73f53123ca3153d112df01eed"
server_key_version: "encryptionkey@0"
I20260620 11:35:46.020536 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.020349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:39643 (local address 127.0.0.1:37844)
0620 11:35:46.020467 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:46.020501 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.189:39643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 11:35:46.020915 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.020794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:39643 (local address 127.0.0.1:37848)
0620 11:35:46.020854 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:46.020889 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.189:39643: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 11:35:46.021013 16399 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.189:39643: Network error: Client connection negotiation failed: client connection to 127.2.243.189:39643: connect: Connection refused (error 111)
I20260620 11:35:46.022082 16406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 11:35:46.022996 16421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:46.023204 16406 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:35:46.023309 16406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-1/wal
uuid: "92184a70a3304f82aa3c183ce425fe61"
format_stamp: "Formatted at 2026-06-20 11:35:46 on dist-test-slave-ctbw"
server_key: "5e06985f16e1fbc9d98df48cd35dc4cf"
server_key_iv: "d03551b73f53123ca3153d112df01eed"
server_key_version: "encryptionkey@0"
I20260620 11:35:46.023375 16406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:46.036020 16406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:46.038103 16406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:46.038239 16406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:46.042667 16406 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.189:39643
I20260620 11:35:46.042693 16473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.189:39643 every 8 connection(s)
I20260620 11:35:46.043268 16406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-1/data/info.pb
I20260620 11:35:46.044250 16474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:35:46.045886 16474 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36401 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 11:35:46 dist-test-slave-ctbw krb5kdc[16318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955346, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.189@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:35:46.053041 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 16406
I20260620 11:35:46.053189 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-1/wal/instance
I20260620 11:35:46.053433 3022 external_mini_cluster.cc:1428] Setting key 742cb2753ccbd1e3f3a7dea6f977eee5
I20260620 11:35:46.055945 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.046675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33934 (local address 127.2.243.190:36401)
0620 11:35:46.046743 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:46.046745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:46.046962 (+ 217us) server_negotiation.cc:407] Connection header received
0620 11:35:46.047465 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:46.047466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:46.047479 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:46.047550 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:46.048181 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.048751 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.049838 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.050079 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.050149 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:46.052525 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:46.052527 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:46.052527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:46.052535 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:46.054138 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.054668 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.054670 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.054670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.054743 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.055093 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.055093 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.055094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.055217 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:46.055272 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:46.055660 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 11:35:46.055812 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 11:35:46.056077 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.046603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:36401 (local address 127.0.0.1:33934)
0620 11:35:46.046795 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:46.046838 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:46.047330 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:46.047635 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:46.047639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:46.047895 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:46.048100 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:46.048104 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.048870 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.048872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:46.049744 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:46.049746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.050163 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.050164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:46.050273 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:46.050939 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:46.050948 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:46.052385 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:46.054250 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:46.054259 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:46.054262 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:46.054563 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:46.054819 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:46.054820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:46.054820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:46.055023 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:46.055286 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:46.055293 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:46.055487 (+ 194us) client_negotiation.cc:769] Sending connection context
0620 11:35:46.055615 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 11:35:46.055814 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260620 11:35:46.056635 16474 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.243.188@KRBTEST.COM; defaulting to no policy
Jun 20 11:35:46 dist-test-slave-ctbw krb5kdc[16318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955346, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.189@KRBTEST.COM for kudu/127.2.243.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.188@KRBTEST.COM" created.
I20260620 11:35:46.064692 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.056961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:39643 (local address 127.0.0.1:37852)
0620 11:35:46.057006 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:46.057016 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:46.057059 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:46.057371 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:46.057372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:46.057527 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:46.057577 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:46.057578 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.058480 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.058482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:46.059076 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:46.059078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.059582 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.059583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:46.059655 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:46.060137 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:46.060142 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:46.061476 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:46.063493 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:46.063494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:46.063495 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:46.063682 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:46.063892 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:46.063893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:46.063894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:46.063958 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:46.064267 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:46.064268 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:46.064316 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 11:35:46.064456 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 11:35:46.064571 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 11:35:46.064778 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.057040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37852 (local address 127.2.243.189:39643)
0620 11:35:46.057228 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:46.057232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:46.057250 (+ 18us) server_negotiation.cc:407] Connection header received
0620 11:35:46.057267 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:46.057269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:46.057279 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:46.057359 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:46.057660 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.058365 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.059166 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.059491 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.059585 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:46.061567 (+ 1982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:46.061578 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:46.061578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:46.061585 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:46.063395 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.063762 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.063766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.063767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.063825 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.064028 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.064029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.064029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.064189 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:46.064281 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:46.064494 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 11:35:46.064650 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":162,"thread_start_us":114,"threads_started":1}
I20260620 11:35:46.065424 16474 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:35:46.065873 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.065722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:39461 (local address 127.0.0.1:57838)
0620 11:35:46.065791 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:46.065845 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 11:35:46.066039 16422 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.243.188:39461: connect: Connection refused (error 111)
I20260620 11:35:46.066627 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.066482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:39461 (local address 127.0.0.1:57844)
0620 11:35:46.066559 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:46.066590 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 11:35:46.066891 16474 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.188:39461: Network error: Client connection negotiation failed: client connection to 127.2.243.188:39461: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.188.keytab.
Entry for principal kudu/127.2.243.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.188.keytab.
I20260620 11:35:46.080031 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.188:39461
--webserver_interface=127.2.243.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.188.keytab
--principal=kudu/127.2.243.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:44649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:36401,127.2.243.189:39643,127.2.243.188:39461
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:35:46.112197 16399 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } attempt: 3
I20260620 11:35:46.116161 16474 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } attempt: 1
I20260620 11:35:46.116679 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.116514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:39461 (local address 127.0.0.1:57848)
0620 11:35:46.116587 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:46.116631 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
Jun 20 11:35:46 dist-test-slave-ctbw krb5kdc[16318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955345, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for kudu/127.2.243.189@KRBTEST.COM
I20260620 11:35:46.117071 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.116963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:39461 (local address 127.0.0.1:57858)
0620 11:35:46.117019 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:46.117042 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 11:35:46.117149 16474 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.188:39461: Network error: Client connection negotiation failed: client connection to 127.2.243.188:39461: connect: Connection refused (error 111)
I20260620 11:35:46.120219 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.112746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:39643 (local address 127.0.0.1:37856)
0620 11:35:46.112808 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:46.112829 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:46.112903 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:46.113066 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:46.113067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:46.113253 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:46.113317 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:46.113319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.114070 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.114071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:46.114606 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:46.114607 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.115091 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.115092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:46.115179 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:46.115708 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:46.115713 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:46.117283 (+ 1570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:46.119112 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:46.119113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:46.119113 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:46.119300 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:46.119529 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:46.119530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:46.119531 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:46.119594 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:46.119866 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:46.119867 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:46.119917 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 11:35:46.120004 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 11:35:46.120122 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 11:35:46.120216 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.112816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37856 (local address 127.2.243.189:39643)
0620 11:35:46.112896 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:46.112898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:46.112910 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:35:46.112970 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:46.112971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:46.112990 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:46.113060 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:46.113410 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.113927 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.114705 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.114996 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.115081 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:46.117391 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:46.117393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:46.117393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:46.117401 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:46.119004 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.119391 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.119393 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.119393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.119453 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.119666 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.119667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.119667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.119790 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:46.119866 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:46.120038 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 11:35:46.120122 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 11:35:46.120644 16399 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:35:46.121028 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.120909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:39461 (local address 127.0.0.1:57860)
0620 11:35:46.120960 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:46.120988 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 11:35:46.121461 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.121396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:39461 (local address 127.0.0.1:57870)
0620 11:35:46.121424 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:46.121447 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260620 11:35:46.121572 16399 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.188:39461: Network error: Client connection negotiation failed: client connection to 127.2.243.188:39461: connect: Connection refused (error 111)
I20260620 11:35:46.152712 16399 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } attempt: 1
I20260620 11:35:46.153299 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.153153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:39461 (local address 127.0.0.1:57878)
0620 11:35:46.153226 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:46.153269 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 11:35:46.153739 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.153650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:39461 (local address 127.0.0.1:57880)
0620 11:35:46.153698 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:46.153723 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 11:35:46.153870 16399 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.188:39461: Network error: Client connection negotiation failed: client connection to 127.2.243.188:39461: connect: Connection refused (error 111)
W20260620 11:35:46.169175 16481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:46.169485 16481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:46.169531 16481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:46.170933 16481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:35:46.171046 16481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:46.171074 16481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:46.171111 16481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:35:46.171144 16481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:35:46.172995 16481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:44649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:36401,127.2.243.189:39643,127.2.243.188:39461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.188.keytab
--principal=kudu/127.2.243.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.188:39461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.243.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:46.173360 16481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:46.173645 16481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:46.179265 16489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:46.179260 16487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:46.179261 16486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:46.179752 16481 server_base.cc:1034] running on GCE node
I20260620 11:35:46.180073 16481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:46.180487 16481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:46.181699 16481 hybrid_clock.cc:648] HybridClock initialized: now 1781955346181686 us; error 51 us; skew 500 ppm
Jun 20 11:35:46 dist-test-slave-ctbw krb5kdc[16318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955346, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:46.184574 16481 init.cc:377] Logged in from keytab as kudu/127.2.243.188@KRBTEST.COM (short username kudu)
I20260620 11:35:46.185112 16481 webserver.cc:460] Webserver started at http://127.2.243.188:36305/ using document root <none> and password file <none>
I20260620 11:35:46.185261 16474 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } attempt: 2
I20260620 11:35:46.185360 16481 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:46.185424 16481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:35:46.185560 16481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:35:46.185729 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.185583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:39461 (local address 127.0.0.1:57886)
0620 11:35:46.185652 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:46.185687 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 11:35:46.186103 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.186010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:39461 (local address 127.0.0.1:57898)
0620 11:35:46.186056 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:46.186080 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:39461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 11:35:46.186223 16474 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.188:39461: Network error: Client connection negotiation failed: client connection to 127.2.243.188:39461: connect: Connection refused (error 111)
I20260620 11:35:46.186839 16481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-2/data/instance:
uuid: "1eff7808b27e40dd82799fe2875a298e"
format_stamp: "Formatted at 2026-06-20 11:35:46 on dist-test-slave-ctbw"
server_key: "3dda0c4ed249c8d91ac4ecaf678a3c15"
server_key_iv: "3148ced113635b72171e4640913fef51"
server_key_version: "encryptionkey@0"
I20260620 11:35:46.187304 16481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-2/wal/instance:
uuid: "1eff7808b27e40dd82799fe2875a298e"
format_stamp: "Formatted at 2026-06-20 11:35:46 on dist-test-slave-ctbw"
server_key: "3dda0c4ed249c8d91ac4ecaf678a3c15"
server_key_iv: "3148ced113635b72171e4640913fef51"
server_key_version: "encryptionkey@0"
I20260620 11:35:46.188702 16481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:35:46.189515 16496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:46.189704 16481 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:35:46.189829 16481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-2/wal
uuid: "1eff7808b27e40dd82799fe2875a298e"
format_stamp: "Formatted at 2026-06-20 11:35:46 on dist-test-slave-ctbw"
server_key: "3dda0c4ed249c8d91ac4ecaf678a3c15"
server_key_iv: "3148ced113635b72171e4640913fef51"
server_key_version: "encryptionkey@0"
I20260620 11:35:46.189910 16481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:46.204550 16481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:46.206652 16481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:46.206840 16481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:46.211040 16481 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.188:39461
I20260620 11:35:46.211226 16548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.188:39461 every 8 connection(s)
I20260620 11:35:46.211520 16481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-2/data/info.pb
I20260620 11:35:46.212415 16549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:35:46.214061 16549 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36401 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:35:46.215734 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 16481
I20260620 11:35:46.215866 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955192827595-3022-0/minicluster-data/master-2/wal/instance
I20260620 11:35:46.216087 3022 external_mini_cluster.cc:1428] Setting key 17f02664f863e2f330eec6854da0163f
Jun 20 11:35:46 dist-test-slave-ctbw krb5kdc[16318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955346, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.188@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
Jun 20 11:35:46 dist-test-slave-ctbw krb5kdc[16318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.188@KRBTEST.COM
I20260620 11:35:46.226049 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.214899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33948 (local address 127.2.243.190:36401)
0620 11:35:46.214961 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:46.214964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:46.215152 (+ 188us) server_negotiation.cc:407] Connection header received
0620 11:35:46.215791 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:46.215793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:46.215845 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:46.215949 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:46.217043 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.217822 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.219243 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.219531 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.219608 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:46.222495 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:46.222497 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:46.222497 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:46.222503 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:46.224545 (+ 2042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.225035 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.225036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.225037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.225103 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.225363 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.225364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.225364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.225493 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:46.225569 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:46.225870 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 11:35:46.225956 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 11:35:46.226234 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.214807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:36401 (local address 127.0.0.1:33948)
0620 11:35:46.215017 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:46.215065 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:46.215604 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:46.216076 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:46.216081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:46.216572 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:46.216940 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:46.216947 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.217901 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.217902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:46.219138 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:46.219141 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.219622 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.219623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:46.219748 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:46.220498 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:46.220507 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:46.222404 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:46.224679 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:46.224686 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:46.224689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:46.224955 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:46.225172 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:46.225173 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:46.225173 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:46.225293 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:46.225560 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:46.225567 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:46.225712 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 11:35:46.225826 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 11:35:46.225991 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":127,"threads_started":1}
I20260620 11:35:46.226950 16549 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:35:46.227082 16553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.216581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57900 (local address 127.2.243.188:39461)
0620 11:35:46.216770 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:46.216777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:46.216820 (+ 43us) server_negotiation.cc:407] Connection header received
0620 11:35:46.216866 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:46.216868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:46.216893 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:46.216982 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:46.217725 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.219164 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.220050 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.220401 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.220485 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:46.223271 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:46.223280 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:46.223280 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:46.223296 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:46.225224 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.225659 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.225662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.225663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.225728 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.225969 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.225970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.225971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.226167 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:46.226355 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:46.226844 (+ 489us) server_negotiation.cc:299] Negotiation successful
0620 11:35:46.226984 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":120,"threads_started":1}
I20260620 11:35:46.229022 16399 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } attempt: 2
Jun 20 11:35:46 dist-test-slave-ctbw krb5kdc[16318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955346, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.188@KRBTEST.COM for kudu/127.2.243.189@KRBTEST.COM
Jun 20 11:35:46 dist-test-slave-ctbw krb5kdc[16318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
Jun 20 11:35:46 dist-test-slave-ctbw krb5kdc[16318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955345, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for kudu/127.2.243.188@KRBTEST.COM
I20260620 11:35:46.235253 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.227331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.2.243.189:39643)
0620 11:35:46.227378 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:46.227381 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:46.227391 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:35:46.227417 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:46.227418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:46.227429 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:46.227496 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:46.227833 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.228697 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.229496 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.229789 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.229849 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:46.232179 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:46.232181 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:46.232182 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:46.232191 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:46.233839 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.234231 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.234232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.234232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.234303 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.234583 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.234584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.234584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.234728 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:46.234812 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:46.235047 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 11:35:46.235141 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:35:46.235275 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.227244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:39643 (local address 127.0.0.1:37862)
0620 11:35:46.227277 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:46.227288 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:46.227329 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:46.227513 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:46.227514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:46.227656 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:46.227718 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:46.227720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.228791 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.228792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:46.229395 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:46.229398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.229905 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.229906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:46.229958 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:46.230404 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:46.230409 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:46.232071 (+ 1662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:46.233936 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:46.233937 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:46.233937 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:46.234139 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:46.234417 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:46.234418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:46.234419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:46.234482 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:46.234814 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:46.234815 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:46.234894 (+ 79us) client_negotiation.cc:769] Sending connection context
0620 11:35:46.235010 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 11:35:46.235140 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260620 11:35:46.235940 16549 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:35:46.237839 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.228053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33962 (local address 127.2.243.190:36401)
0620 11:35:46.228085 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:46.228087 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:46.228094 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:35:46.228162 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:46.228162 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:46.228170 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:46.228211 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:46.228833 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.229656 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.230741 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.231100 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.231175 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:46.234136 (+ 2961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:46.234137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:46.234138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:46.234144 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:46.236079 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.236514 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.236516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.236516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.236583 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.236839 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.236841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.236841 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.236985 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:46.237185 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:46.237629 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 11:35:46.237719 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 11:35:46.240437 16401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.229335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:39461 (local address 127.0.0.1:57902)
0620 11:35:46.229380 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:46.229390 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:46.229425 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:46.229551 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:46.229552 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:46.229710 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:46.229744 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:46.229745 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.231048 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.231050 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:46.231584 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:46.231586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.232018 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.232019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:46.232077 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:46.232568 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:46.232573 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:46.234612 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:46.239270 (+ 4658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:46.239271 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:46.239272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:46.239520 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:46.239738 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:46.239738 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:46.239739 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:46.239815 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:46.240095 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:46.240096 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:46.240143 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 11:35:46.240207 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 11:35:46.240306 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 11:35:46.240734 16553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.229396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57902 (local address 127.2.243.188:39461)
0620 11:35:46.229432 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:46.229434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:46.229468 (+ 34us) server_negotiation.cc:407] Connection header received
0620 11:35:46.229487 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:46.229488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:46.229498 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:46.229548 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:46.229815 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.230952 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.231671 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.231940 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.232018 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:46.234695 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:46.234696 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:46.234697 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:46.234705 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:46.237609 (+ 2904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.239604 (+ 1995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.239605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.239605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.239666 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.239886 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.239887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.239888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.240012 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:46.240071 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:46.240554 (+ 483us) server_negotiation.cc:299] Negotiation successful
0620 11:35:46.240636 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Jun 20 11:35:46 dist-test-slave-ctbw krb5kdc[16318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955346, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.188@KRBTEST.COM for kudu/127.2.243.188@KRBTEST.COM
I20260620 11:35:46.242518 16399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458: Bootstrap starting.
I20260620 11:35:46.243413 16399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458: Neither blocks nor log segments found. Creating new log.
I20260620 11:35:46.243757 16399 log.cc:824] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458: Log is configured to *not* fsync() on all Append() calls
I20260620 11:35:46.244709 16399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458: No bootstrap required, opened a new log
Jun 20 11:35:46 dist-test-slave-ctbw krb5kdc[16318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.189@KRBTEST.COM
I20260620 11:35:46.246039 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.236402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57904 (local address 127.2.243.188:39461)
0620 11:35:46.236588 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:46.236590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:46.236598 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:35:46.236618 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:46.236619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:46.236675 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:46.236733 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:46.237853 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.239231 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.240419 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.240742 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.240811 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:46.242719 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:46.242720 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:46.242721 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:46.242727 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:46.244629 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.245021 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.245022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.245022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.245086 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.245359 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.245359 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.245360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.245520 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:46.245614 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:46.245781 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 11:35:46.245886 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":123,"threads_started":1}
I20260620 11:35:46.246048 16551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.236334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:39461 (local address 127.0.0.1:57904)
0620 11:35:46.236383 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:46.236395 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:46.236437 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:46.236807 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:46.236808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:46.237697 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:46.237737 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:46.237739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.239447 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.239448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:46.240337 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:46.240339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.240858 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.240859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:46.240911 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:46.241385 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:46.241391 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:46.242644 (+ 1253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:46.244739 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:46.244740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:46.244741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:46.244928 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:46.245158 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:46.245159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:46.245160 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:46.245219 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:46.245605 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:46.245606 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:46.245649 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 11:35:46.245756 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 11:35:46.245945 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":73}
I20260620 11:35:46.247222 16399 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f604dbffb44b0ca15d06c0c0696458" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36401 } } peers { permanent_uuid: "92184a70a3304f82aa3c183ce425fe61" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } } peers { permanent_uuid: "1eff7808b27e40dd82799fe2875a298e" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } }
I20260620 11:35:46.247375 16399 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:35:46.247422 16399 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3f604dbffb44b0ca15d06c0c0696458, State: Initialized, Role: FOLLOWER
I20260620 11:35:46.247586 16399 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f604dbffb44b0ca15d06c0c0696458" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36401 } } peers { permanent_uuid: "92184a70a3304f82aa3c183ce425fe61" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } } peers { permanent_uuid: "1eff7808b27e40dd82799fe2875a298e" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } }
I20260620 11:35:46.247854 16549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e: Bootstrap starting.
I20260620 11:35:46.247964 16399 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:35:46.247901 16556 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f604dbffb44b0ca15d06c0c0696458" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36401 } } peers { permanent_uuid: "92184a70a3304f82aa3c183ce425fe61" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } } peers { permanent_uuid: "1eff7808b27e40dd82799fe2875a298e" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } } }
I20260620 11:35:46.248147 16556 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:35:46.248708 16549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e: Neither blocks nor log segments found. Creating new log.
I20260620 11:35:46.248965 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.239211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37874 (local address 127.2.243.189:39643)
0620 11:35:46.239392 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:46.239394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:46.239404 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:35:46.239456 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:46.239457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:46.239467 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:46.239553 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:46.240462 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.241118 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.242485 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.242836 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.242976 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:46.245845 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:46.245847 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:46.245847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:46.245856 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:46.247647 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.248038 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.248040 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.248040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.248097 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.248335 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.248337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.248337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.248493 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:46.248706 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:46.248755 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 11:35:46.248859 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260620 11:35:46.249055 16549 log.cc:824] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e: Log is configured to *not* fsync() on all Append() calls
I20260620 11:35:46.250032 16549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e: No bootstrap required, opened a new log
W20260620 11:35:46.251652 16568 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:35:46.251765 16568 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:35:46.251978 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:35:46.252230 16549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f604dbffb44b0ca15d06c0c0696458" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36401 } } peers { permanent_uuid: "92184a70a3304f82aa3c183ce425fe61" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } } peers { permanent_uuid: "1eff7808b27e40dd82799fe2875a298e" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } }
I20260620 11:35:46.252393 16549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:35:46.252440 16549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eff7808b27e40dd82799fe2875a298e, State: Initialized, Role: FOLLOWER
I20260620 11:35:46.252573 16549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f604dbffb44b0ca15d06c0c0696458" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36401 } } peers { permanent_uuid: "92184a70a3304f82aa3c183ce425fe61" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } } peers { permanent_uuid: "1eff7808b27e40dd82799fe2875a298e" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } }
I20260620 11:35:46.252852 16570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f604dbffb44b0ca15d06c0c0696458" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36401 } } peers { permanent_uuid: "92184a70a3304f82aa3c183ce425fe61" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } } peers { permanent_uuid: "1eff7808b27e40dd82799fe2875a298e" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } } }
I20260620 11:35:46.252921 16549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:35:46.252993 16570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e [sys.catalog]: This master's current role is: FOLLOWER
W20260620 11:35:46.255141 16587 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:35:46.255256 16587 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:35:46.252080 16331 master_runner.cc:186] Error getting master registration for 127.2.243.189:39643: OK, instance_id { permanent_uuid: "92184a70a3304f82aa3c183ce425fe61" instance_seqno: 1781955346041420 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 11:35:46.255487 16331 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 11:35:46.256982 16481 master_runner.cc:186] Error getting master registration for 127.2.243.189:39643: OK, instance_id { permanent_uuid: "92184a70a3304f82aa3c183ce425fe61" instance_seqno: 1781955346041420 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 11:35:46.259307 16481 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 11:35:46.263429 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.253839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57908 (local address 127.2.243.188:39461)
0620 11:35:46.254554 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:46.254556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:46.254567 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:35:46.254796 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:46.254797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:46.254809 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:46.255711 (+ 902us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:46.255744 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.256722 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.257578 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.257823 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.257918 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:46.259656 (+ 1738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:46.259658 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:46.259658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:46.259666 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:46.261257 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.262057 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.262059 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.262059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.262127 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.262648 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.262649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.262649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.262816 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:46.262995 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:46.263239 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 11:35:46.263336 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693}
I20260620 11:35:46.265131 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.253580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37890 (local address 127.2.243.189:39643)
0620 11:35:46.253634 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:46.253636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:46.254037 (+ 401us) server_negotiation.cc:407] Connection header received
0620 11:35:46.254168 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:46.254169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:46.254179 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:46.254208 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:46.255251 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.255979 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.259612 (+ 3633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.259928 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.259971 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:46.261415 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:46.261416 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:46.261417 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:46.261424 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:46.263286 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.263727 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.263729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.263729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.263802 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.264440 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.264441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.264442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.264589 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:46.264664 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:46.264952 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 11:35:46.265040 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 11:35:46.272985 16402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.259208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33964 (local address 127.2.243.190:36401)
0620 11:35:46.259255 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:46.259258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:46.264080 (+ 4822us) server_negotiation.cc:407] Connection header received
0620 11:35:46.264188 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:46.264190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:46.264202 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:46.264234 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:46.264911 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.265739 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.267536 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.267877 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.267927 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:46.269607 (+ 1680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:46.269608 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:46.269609 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:46.269617 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:46.271531 (+ 1914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.271987 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.271988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.271989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.272050 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.272279 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.272280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.272280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.272427 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:46.272522 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:46.272786 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 11:35:46.272876 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:35:46.277386 16474 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } attempt: 3
Jun 20 11:35:46 dist-test-slave-ctbw krb5kdc[16318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955346, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.189@KRBTEST.COM for kudu/127.2.243.188@KRBTEST.COM
I20260620 11:35:46.286899 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.277729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:39461 (local address 127.0.0.1:57920)
0620 11:35:46.277891 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:46.277902 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:46.277938 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:46.278285 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:46.278286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:46.278472 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:46.278525 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:46.278526 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.279680 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.279681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:46.280499 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:46.280501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.281094 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.281096 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:46.281164 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:46.281680 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:46.281687 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:46.283397 (+ 1710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:46.285614 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:46.285615 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:46.285616 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:46.285876 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:46.286114 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:46.286115 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:46.286116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:46.286187 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:46.286496 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:46.286497 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:46.286551 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 11:35:46.286627 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 11:35:46.286738 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260620 11:35:46.286911 16554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:46.278012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57920 (local address 127.2.243.188:39461)
0620 11:35:46.278144 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:46.278146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:46.278152 (+ 6us) server_negotiation.cc:407] Connection header received
0620 11:35:46.278174 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:46.278175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:46.278184 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:46.278234 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:46.278618 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.279591 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.280601 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:46.280946 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:46.281024 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:46.283499 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:46.283500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:46.283501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:46.283511 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:46.285507 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.285963 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.285965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.285965 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.286034 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:46.286259 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:46.286260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:46.286260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:46.286420 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:46.286497 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:46.286673 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 11:35:46.286774 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260620 11:35:46.289186 16474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61: Bootstrap starting.
I20260620 11:35:46.290066 16474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61: Neither blocks nor log segments found. Creating new log.
I20260620 11:35:46.290400 16474 log.cc:824] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61: Log is configured to *not* fsync() on all Append() calls
I20260620 11:35:46.291477 16474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61: No bootstrap required, opened a new log
I20260620 11:35:46.294059 16474 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f604dbffb44b0ca15d06c0c0696458" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36401 } } peers { permanent_uuid: "92184a70a3304f82aa3c183ce425fe61" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } } peers { permanent_uuid: "1eff7808b27e40dd82799fe2875a298e" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } }
I20260620 11:35:46.294317 16474 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:35:46.294411 16474 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92184a70a3304f82aa3c183ce425fe61, State: Initialized, Role: FOLLOWER
I20260620 11:35:46.294637 16474 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f604dbffb44b0ca15d06c0c0696458" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36401 } } peers { permanent_uuid: "92184a70a3304f82aa3c183ce425fe61" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } } peers { permanent_uuid: "1eff7808b27e40dd82799fe2875a298e" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } }
I20260620 11:35:46.295145 16474 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 11:35:46.297312 16602 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:35:46.297374 16602 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:35:46.299592 16406 master_runner.cc:263] No leader master found from master 92184a70a3304f82aa3c183ce425fe61
I20260620 11:35:46.299649 16406 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 11:35:46.299793 16591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f604dbffb44b0ca15d06c0c0696458" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36401 } } peers { permanent_uuid: "92184a70a3304f82aa3c183ce425fe61" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } } peers { permanent_uuid: "1eff7808b27e40dd82799fe2875a298e" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } } }
I20260620 11:35:46.299971 16591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 11:35:46.379392 16352 tablet.cc:2369] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 11:35:46.457573 16502 tablet.cc:2369] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:35:46.491101 16556 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:35:46.491247 16556 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f604dbffb44b0ca15d06c0c0696458" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36401 } } peers { permanent_uuid: "92184a70a3304f82aa3c183ce425fe61" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } } peers { permanent_uuid: "1eff7808b27e40dd82799fe2875a298e" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } }
I20260620 11:35:46.491591 16556 leader_election.cc:288] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92184a70a3304f82aa3c183ce425fe61 (127.2.243.189:39643), 1eff7808b27e40dd82799fe2875a298e (127.2.243.188:39461)
I20260620 11:35:46.491894 16523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3f604dbffb44b0ca15d06c0c0696458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eff7808b27e40dd82799fe2875a298e" is_pre_election: true
I20260620 11:35:46.491914 16448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3f604dbffb44b0ca15d06c0c0696458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92184a70a3304f82aa3c183ce425fe61" is_pre_election: true
I20260620 11:35:46.492067 16448 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3f604dbffb44b0ca15d06c0c0696458 in term 0.
I20260620 11:35:46.492084 16523 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3f604dbffb44b0ca15d06c0c0696458 in term 0.
I20260620 11:35:46.492275 16350 leader_election.cc:302] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92184a70a3304f82aa3c183ce425fe61, d3f604dbffb44b0ca15d06c0c0696458; no voters:
I20260620 11:35:46.492421 16556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:35:46.492475 16556 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:35:46.492498 16556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:35:46.493402 16556 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f604dbffb44b0ca15d06c0c0696458" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36401 } } peers { permanent_uuid: "92184a70a3304f82aa3c183ce425fe61" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } } peers { permanent_uuid: "1eff7808b27e40dd82799fe2875a298e" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } }
I20260620 11:35:46.493605 16556 leader_election.cc:288] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [CANDIDATE]: Term 1 election: Requested vote from peers 92184a70a3304f82aa3c183ce425fe61 (127.2.243.189:39643), 1eff7808b27e40dd82799fe2875a298e (127.2.243.188:39461)
I20260620 11:35:46.493785 16448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3f604dbffb44b0ca15d06c0c0696458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92184a70a3304f82aa3c183ce425fe61"
I20260620 11:35:46.493832 16523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d3f604dbffb44b0ca15d06c0c0696458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1eff7808b27e40dd82799fe2875a298e"
I20260620 11:35:46.493903 16448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:35:46.493974 16523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:35:46.494633 16448 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f604dbffb44b0ca15d06c0c0696458 in term 1.
I20260620 11:35:46.494937 16350 leader_election.cc:302] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92184a70a3304f82aa3c183ce425fe61, d3f604dbffb44b0ca15d06c0c0696458; no voters:
I20260620 11:35:46.494988 16523 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3f604dbffb44b0ca15d06c0c0696458 in term 1.
I20260620 11:35:46.495070 16556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:35:46.495244 16556 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [term 1 LEADER]: Becoming Leader. State: Replica: d3f604dbffb44b0ca15d06c0c0696458, State: Running, Role: LEADER
I20260620 11:35:46.495353 16556 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f604dbffb44b0ca15d06c0c0696458" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36401 } } peers { permanent_uuid: "92184a70a3304f82aa3c183ce425fe61" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } } peers { permanent_uuid: "1eff7808b27e40dd82799fe2875a298e" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } }
I20260620 11:35:46.495864 16604 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3f604dbffb44b0ca15d06c0c0696458. Latest consensus state: current_term: 1 leader_uuid: "d3f604dbffb44b0ca15d06c0c0696458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f604dbffb44b0ca15d06c0c0696458" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36401 } } peers { permanent_uuid: "92184a70a3304f82aa3c183ce425fe61" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } } peers { permanent_uuid: "1eff7808b27e40dd82799fe2875a298e" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } } }
I20260620 11:35:46.495965 16604 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [sys.catalog]: This master's current role is: LEADER
I20260620 11:35:46.496178 16606 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:35:46.496451 16606 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:35:46.498003 16448 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61 [term 1 FOLLOWER]: Refusing update from remote peer d3f604dbffb44b0ca15d06c0c0696458: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:35:46.498013 16523 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e [term 1 FOLLOWER]: Refusing update from remote peer d3f604dbffb44b0ca15d06c0c0696458: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:35:46.498338 16556 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1eff7808b27e40dd82799fe2875a298e" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:35:46.498476 16604 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92184a70a3304f82aa3c183ce425fe61" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:35:46.499799 16570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3f604dbffb44b0ca15d06c0c0696458. Latest consensus state: current_term: 1 leader_uuid: "d3f604dbffb44b0ca15d06c0c0696458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f604dbffb44b0ca15d06c0c0696458" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36401 } } peers { permanent_uuid: "92184a70a3304f82aa3c183ce425fe61" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } } peers { permanent_uuid: "1eff7808b27e40dd82799fe2875a298e" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } } }
I20260620 11:35:46.499805 16591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3f604dbffb44b0ca15d06c0c0696458. Latest consensus state: current_term: 1 leader_uuid: "d3f604dbffb44b0ca15d06c0c0696458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f604dbffb44b0ca15d06c0c0696458" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36401 } } peers { permanent_uuid: "92184a70a3304f82aa3c183ce425fe61" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } } peers { permanent_uuid: "1eff7808b27e40dd82799fe2875a298e" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } } }
I20260620 11:35:46.499908 16570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:35:46.499917 16591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:35:46.500944 16556 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3f604dbffb44b0ca15d06c0c0696458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f604dbffb44b0ca15d06c0c0696458" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36401 } } peers { permanent_uuid: "92184a70a3304f82aa3c183ce425fe61" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } } peers { permanent_uuid: "1eff7808b27e40dd82799fe2875a298e" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } } }
I20260620 11:35:46.501051 16556 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [sys.catalog]: This master's current role is: LEADER
I20260620 11:35:46.501097 16556 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d3f604dbffb44b0ca15d06c0c0696458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f604dbffb44b0ca15d06c0c0696458" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36401 } } peers { permanent_uuid: "92184a70a3304f82aa3c183ce425fe61" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } } peers { permanent_uuid: "1eff7808b27e40dd82799fe2875a298e" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } } }
I20260620 11:35:46.501140 16556 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458 [sys.catalog]: This master's current role is: LEADER
I20260620 11:35:46.501159 16570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3f604dbffb44b0ca15d06c0c0696458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f604dbffb44b0ca15d06c0c0696458" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36401 } } peers { permanent_uuid: "92184a70a3304f82aa3c183ce425fe61" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } } peers { permanent_uuid: "1eff7808b27e40dd82799fe2875a298e" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } } }
I20260620 11:35:46.501237 16570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:35:46.501293 16607 mvcc.cc:205] Tried to move back new op lower bound from 7298889099253768192 to 7298889099245879296. Current Snapshot: MvccSnapshot[applied={T|T < 7298889099253768192}]
I20260620 11:35:46.501298 16591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d3f604dbffb44b0ca15d06c0c0696458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3f604dbffb44b0ca15d06c0c0696458" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36401 } } peers { permanent_uuid: "92184a70a3304f82aa3c183ce425fe61" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 39643 } } peers { permanent_uuid: "1eff7808b27e40dd82799fe2875a298e" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 39461 } } }
I20260620 11:35:46.501377 16591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:35:46.501516 16609 mvcc.cc:205] Tried to move back new op lower bound from 7298889099253768192 to 7298889099245879296. Current Snapshot: MvccSnapshot[applied={T|T < 7298889099253768192}]
I20260620 11:35:46.501634 16608 mvcc.cc:205] Tried to move back new op lower bound from 7298889099253768192 to 7298889099245879296. Current Snapshot: MvccSnapshot[applied={T|T < 7298889099253768192}]
I20260620 11:35:46.502151 16606 catalog_manager.cc:1347] Generated new cluster ID: 9a80fde4974e4094863a4861cf744d96
I20260620 11:35:46.502197 16606 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:35:46.517335 16606 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:35:46.518051 16606 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:35:46.521865 16606 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3f604dbffb44b0ca15d06c0c0696458: Generated new TSK 0
I20260620 11:35:46.522133 16606 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:35:47.255724 16587 catalog_manager.cc:1259] Loaded cluster ID: 9a80fde4974e4094863a4861cf744d96
I20260620 11:35:47.255801 16587 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e: loading cluster ID for follower catalog manager: success
I20260620 11:35:47.256659 16587 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e: acquiring CA information for follower catalog manager: success
I20260620 11:35:47.256973 16587 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1eff7808b27e40dd82799fe2875a298e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 11:35:47.297786 16602 catalog_manager.cc:1259] Loaded cluster ID: 9a80fde4974e4094863a4861cf744d96
I20260620 11:35:47.297886 16602 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61: loading cluster ID for follower catalog manager: success
I20260620 11:35:47.298693 16602 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61: acquiring CA information for follower catalog manager: success
I20260620 11:35:47.299033 16602 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 92184a70a3304f82aa3c183ce425fe61: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 11:35:47.576795 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 16331
I20260620 11:35:47.582145 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 16406
I20260620 11:35:47.587097 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 16481
2026-06-20T11:35:47Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2995 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:35:47 dist-test-slave-ctbw krb5kdc[16622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:35:47 dist-test-slave-ctbw krb5kdc[16622](info): set up 2 sockets
Jun 20 11:35:47 dist-test-slave-ctbw krb5kdc[16622](info): commencing operation
krb5kdc: starting...
W20260620 11:35:48.619063 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:35:48 dist-test-slave-ctbw krb5kdc[16622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:35:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:35:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:35:48.735991 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:42431
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:42953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:42431
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:35:48.816744 16635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:48.817014 16635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:48.817077 16635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:48.818280 16635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:35:48.818356 16635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:48.818379 16635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:48.818400 16635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:35:48.818430 16635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:35:48.819958 16635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:42953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:42431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:42431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:48.820252 16635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:48.820488 16635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:48.825485 16640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:48.825513 16643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:48.825476 16641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:48.825675 16635 server_base.cc:1034] running on GCE node
I20260620 11:35:48.826107 16635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:48.826433 16635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:48.827585 16635 hybrid_clock.cc:648] HybridClock initialized: now 1781955348827571 us; error 31 us; skew 500 ppm
Jun 20 11:35:48 dist-test-slave-ctbw krb5kdc[16622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955348, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:48.830380 16635 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:35:48.830976 16635 webserver.cc:460] Webserver started at http://127.2.243.190:35401/ using document root <none> and password file <none>
I20260620 11:35:48.831238 16635 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:48.831288 16635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:35:48.831444 16635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:35:48.832661 16635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "e705a7baea82450080ab601af376190e"
format_stamp: "Formatted at 2026-06-20 11:35:48 on dist-test-slave-ctbw"
server_key: "9514fff8d9bc35bd1c251bf49a392fd2"
server_key_iv: "7bdcdc076b2b1ab9687410c3d52fb565"
server_key_version: "encryptionkey@0"
I20260620 11:35:48.833060 16635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "e705a7baea82450080ab601af376190e"
format_stamp: "Formatted at 2026-06-20 11:35:48 on dist-test-slave-ctbw"
server_key: "9514fff8d9bc35bd1c251bf49a392fd2"
server_key_iv: "7bdcdc076b2b1ab9687410c3d52fb565"
server_key_version: "encryptionkey@0"
I20260620 11:35:48.834553 16635 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:35:48.835695 16650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:48.835920 16635 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:35:48.836015 16635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "e705a7baea82450080ab601af376190e"
format_stamp: "Formatted at 2026-06-20 11:35:48 on dist-test-slave-ctbw"
server_key: "9514fff8d9bc35bd1c251bf49a392fd2"
server_key_iv: "7bdcdc076b2b1ab9687410c3d52fb565"
server_key_version: "encryptionkey@0"
I20260620 11:35:48.836128 16635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:48.850852 16635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:48.852947 16635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:48.853116 16635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:48.857043 16635 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:42431
I20260620 11:35:48.857121 16702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:42431 every 8 connection(s)
I20260620 11:35:48.857476 16635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:35:48.858388 16703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:35:48.860718 16703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e: Bootstrap starting.
I20260620 11:35:48.860922 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 16635
I20260620 11:35:48.861022 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:35:48.861239 3022 external_mini_cluster.cc:1428] Setting key bf3ed5d2f3961f97360f31deb01305f8
I20260620 11:35:48.861482 16703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e: Neither blocks nor log segments found. Creating new log.
I20260620 11:35:48.861830 16703 log.cc:824] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e: Log is configured to *not* fsync() on all Append() calls
I20260620 11:35:48.862498 16703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e: No bootstrap required, opened a new log
I20260620 11:35:48.864389 16703 raft_consensus.cc:348] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e705a7baea82450080ab601af376190e" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 42431 } }
I20260620 11:35:48.864552 16703 raft_consensus.cc:374] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:35:48.864601 16703 raft_consensus.cc:729] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e705a7baea82450080ab601af376190e, State: Initialized, Role: FOLLOWER
I20260620 11:35:48.864744 16703 consensus_queue.cc:260] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e705a7baea82450080ab601af376190e" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 42431 } }
I20260620 11:35:48.864821 16703 raft_consensus.cc:388] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:35:48.864863 16703 raft_consensus.cc:482] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:35:48.864912 16703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:35:48.865443 16703 raft_consensus.cc:504] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e705a7baea82450080ab601af376190e" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 42431 } }
I20260620 11:35:48.865558 16703 leader_election.cc:302] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e705a7baea82450080ab601af376190e; no voters:
I20260620 11:35:48.865756 16703 leader_election.cc:288] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:35:48.865779 16708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:35:48.865891 16708 raft_consensus.cc:686] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e [term 1 LEADER]: Becoming Leader. State: Replica: e705a7baea82450080ab601af376190e, State: Running, Role: LEADER
I20260620 11:35:48.866026 16703 sys_catalog.cc:564] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:35:48.866034 16708 consensus_queue.cc:237] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e705a7baea82450080ab601af376190e" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 42431 } }
I20260620 11:35:48.866832 16710 sys_catalog.cc:455] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e705a7baea82450080ab601af376190e. Latest consensus state: current_term: 1 leader_uuid: "e705a7baea82450080ab601af376190e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e705a7baea82450080ab601af376190e" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 42431 } } }
I20260620 11:35:48.866932 16710 sys_catalog.cc:458] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e [sys.catalog]: This master's current role is: LEADER
I20260620 11:35:48.867153 16709 sys_catalog.cc:455] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e705a7baea82450080ab601af376190e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e705a7baea82450080ab601af376190e" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 42431 } } }
I20260620 11:35:48.867265 16709 sys_catalog.cc:458] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e [sys.catalog]: This master's current role is: LEADER
I20260620 11:35:48.867431 16712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:35:48.867966 16712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 11:35:48 dist-test-slave-ctbw krb5kdc[16622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:35:48.869707 16712 catalog_manager.cc:1347] Generated new cluster ID: f73a9418b9e14ad0b88b16014731b8d4
I20260620 11:35:48.869763 16712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:35:48.872808 16706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:48.861744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43312 (local address 127.2.243.190:42431)
0620 11:35:48.861943 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:48.861953 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:48.861986 (+ 33us) server_negotiation.cc:407] Connection header received
0620 11:35:48.862429 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:48.862436 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:48.862481 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:48.862648 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:48.863378 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:48.864271 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:48.865281 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:48.865786 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:48.865910 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:48.869040 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:48.869045 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:48.869045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:48.869058 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:48.871147 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:48.871531 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:48.871534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:48.871535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:48.871641 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:48.871857 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:48.871858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:48.871858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:48.872110 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:48.872274 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:48.872398 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 11:35:48.872648 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":85,"threads_started":1}
I20260620 11:35:48.873804 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:35:48.880906 16712 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:35:48.881614 16712 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:35:48.890710 16712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e705a7baea82450080ab601af376190e: Generated new TSK 0
I20260620 11:35:48.890936 16712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:35:48.927795 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 16635
2026-06-20T11:35:48Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1338 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:35:48 dist-test-slave-ctbw krb5kdc[16730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:35:48 dist-test-slave-ctbw krb5kdc[16730](info): set up 2 sockets
Jun 20 11:35:48 dist-test-slave-ctbw krb5kdc[16730](info): commencing operation
krb5kdc: starting...
W20260620 11:35:49.955544 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:35:50 dist-test-slave-ctbw krb5kdc[16730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:35:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:35:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:35:50.073571 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:43247
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:43247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:35:50.155164 16743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:50.155369 16743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:50.155390 16743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:50.156577 16743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:35:50.156625 16743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:50.156638 16743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:50.156652 16743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:35:50.156664 16743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:35:50.158221 16743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:43247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:43247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:50.158479 16743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:50.158737 16743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:50.163951 16751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:50.164103 16743 server_base.cc:1034] running on GCE node
W20260620 11:35:50.163976 16748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:50.163973 16749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:50.164546 16743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:50.164898 16743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:50.166097 16743 hybrid_clock.cc:648] HybridClock initialized: now 1781955350166088 us; error 35 us; skew 500 ppm
Jun 20 11:35:50 dist-test-slave-ctbw krb5kdc[16730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955350, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:50.168999 16743 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:35:50.169546 16743 webserver.cc:460] Webserver started at http://127.2.243.190:46295/ using document root <none> and password file <none>
I20260620 11:35:50.169788 16743 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:50.169858 16743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:35:50.170004 16743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:35:50.171214 16743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "4d3c51a014204e628be7105bd6536a55"
format_stamp: "Formatted at 2026-06-20 11:35:50 on dist-test-slave-ctbw"
server_key: "c80bf89a0991fe682ce46b9d61cd9344"
server_key_iv: "5e2db0a71baa6413d108e0efbbee123f"
server_key_version: "encryptionkey@0"
I20260620 11:35:50.171598 16743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "4d3c51a014204e628be7105bd6536a55"
format_stamp: "Formatted at 2026-06-20 11:35:50 on dist-test-slave-ctbw"
server_key: "c80bf89a0991fe682ce46b9d61cd9344"
server_key_iv: "5e2db0a71baa6413d108e0efbbee123f"
server_key_version: "encryptionkey@0"
I20260620 11:35:50.172930 16743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:35:50.173695 16758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:50.173868 16743 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:35:50.173979 16743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "4d3c51a014204e628be7105bd6536a55"
format_stamp: "Formatted at 2026-06-20 11:35:50 on dist-test-slave-ctbw"
server_key: "c80bf89a0991fe682ce46b9d61cd9344"
server_key_iv: "5e2db0a71baa6413d108e0efbbee123f"
server_key_version: "encryptionkey@0"
I20260620 11:35:50.174062 16743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:50.187700 16743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:50.191020 16743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:50.191354 16743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:50.197580 16743 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:43247
I20260620 11:35:50.197672 16810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:43247 every 8 connection(s)
I20260620 11:35:50.198019 16743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:35:50.198770 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 16743
I20260620 11:35:50.198935 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:35:50.199182 3022 external_mini_cluster.cc:1428] Setting key e221d2b023bbd44206ce41b74be7b96e
I20260620 11:35:50.199584 16811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:35:50.204209 16811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55: Bootstrap starting.
I20260620 11:35:50.205004 16811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55: Neither blocks nor log segments found. Creating new log.
I20260620 11:35:50.205327 16811 log.cc:824] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55: Log is configured to *not* fsync() on all Append() calls
I20260620 11:35:50.206233 16811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55: No bootstrap required, opened a new log
I20260620 11:35:50.208940 16811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c51a014204e628be7105bd6536a55" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43247 } }
I20260620 11:35:50.209158 16811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:35:50.209200 16811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3c51a014204e628be7105bd6536a55, State: Initialized, Role: FOLLOWER
I20260620 11:35:50.209331 16811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c51a014204e628be7105bd6536a55" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43247 } }
I20260620 11:35:50.209422 16811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:35:50.209455 16811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:35:50.209489 16811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:35:50.210059 16811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c51a014204e628be7105bd6536a55" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43247 } }
I20260620 11:35:50.210182 16811 leader_election.cc:302] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d3c51a014204e628be7105bd6536a55; no voters:
I20260620 11:35:50.210361 16811 leader_election.cc:288] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:35:50.210412 16816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:35:50.210579 16816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 1 LEADER]: Becoming Leader. State: Replica: 4d3c51a014204e628be7105bd6536a55, State: Running, Role: LEADER
I20260620 11:35:50.210642 16811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:35:50.210783 16816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c51a014204e628be7105bd6536a55" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43247 } }
I20260620 11:35:50.211258 16816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d3c51a014204e628be7105bd6536a55. Latest consensus state: current_term: 1 leader_uuid: "4d3c51a014204e628be7105bd6536a55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c51a014204e628be7105bd6536a55" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43247 } } }
I20260620 11:35:50.211268 16818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d3c51a014204e628be7105bd6536a55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c51a014204e628be7105bd6536a55" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43247 } } }
I20260620 11:35:50.211419 16816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [sys.catalog]: This master's current role is: LEADER
I20260620 11:35:50.211473 16818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [sys.catalog]: This master's current role is: LEADER
I20260620 11:35:50.211730 16825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:35:50.212245 16825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:35:50.214696 16825 catalog_manager.cc:1347] Generated new cluster ID: f76c64f9d6174de8ae97c7b6b315675e
I20260620 11:35:50.214792 16825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:35:50 dist-test-slave-ctbw krb5kdc[16730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:35:50.219762 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:50.199912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60348 (local address 127.2.243.190:43247)
0620 11:35:50.200293 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:50.200303 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:50.200338 (+ 35us) server_negotiation.cc:407] Connection header received
0620 11:35:50.201306 (+ 968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:50.201337 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:50.201417 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:50.201589 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:50.202753 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.204321 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.205604 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.206263 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.206402 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:50.215446 (+ 9044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:50.215450 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:50.215450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:50.215464 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:50.217813 (+ 2349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:50.218208 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:50.218212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:50.218212 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:50.218337 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:50.218517 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:50.218518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:50.218518 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:50.218865 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:50.219061 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:50.219380 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 11:35:50.219618 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":69,"threads_started":1}
I20260620 11:35:50.224031 16825 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:35:50.224561 16825 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:35:50.234239 16825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55: Generated new TSK 0
I20260620 11:35:50.234467 16825 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
I20260620 11:35:50.265441 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:43247
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:35:50.365770 16837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:50.365998 16837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:50.366041 16837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:50.367282 16837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:50.367349 16837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:50.367399 16837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:35:50.368845 16837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:43247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:50.369128 16837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:50.369366 16837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:50.374428 16843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:50.374442 16842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:50.374634 16837 server_base.cc:1034] running on GCE node
W20260620 11:35:50.374446 16845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:50.374969 16837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:50.375244 16837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:50.376412 16837 hybrid_clock.cc:648] HybridClock initialized: now 1781955350376396 us; error 35 us; skew 500 ppm
Jun 20 11:35:50 dist-test-slave-ctbw krb5kdc[16730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955350, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:50.379138 16837 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:35:50.379675 16837 webserver.cc:460] Webserver started at http://127.2.243.129:37769/ using document root <none> and password file <none>
I20260620 11:35:50.379926 16837 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:50.379997 16837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:35:50.380149 16837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:35:50.381318 16837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "7371fe0ea9e04b2196329df942c10b7d"
format_stamp: "Formatted at 2026-06-20 11:35:50 on dist-test-slave-ctbw"
server_key: "bf3ab1fb9cf881dd48eff4788687eb08"
server_key_iv: "0df0a2f77e7002c3977fa7570d3fd7be"
server_key_version: "encryptionkey@0"
I20260620 11:35:50.381676 16837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "7371fe0ea9e04b2196329df942c10b7d"
format_stamp: "Formatted at 2026-06-20 11:35:50 on dist-test-slave-ctbw"
server_key: "bf3ab1fb9cf881dd48eff4788687eb08"
server_key_iv: "0df0a2f77e7002c3977fa7570d3fd7be"
server_key_version: "encryptionkey@0"
I20260620 11:35:50.383100 16837 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 11:35:50.383895 16852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:50.384078 16837 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:35:50.384177 16837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "7371fe0ea9e04b2196329df942c10b7d"
format_stamp: "Formatted at 2026-06-20 11:35:50 on dist-test-slave-ctbw"
server_key: "bf3ab1fb9cf881dd48eff4788687eb08"
server_key_iv: "0df0a2f77e7002c3977fa7570d3fd7be"
server_key_version: "encryptionkey@0"
I20260620 11:35:50.384254 16837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:50.397177 16837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:50.399637 16837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:50.399804 16837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:50.400058 16837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:35:50.400373 16837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:35:50.400424 16837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:50.400466 16837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:35:50.400490 16837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:50.406366 16837 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:39513
I20260620 11:35:50.406420 16965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:39513 every 8 connection(s)
I20260620 11:35:50.406745 16837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:35:50.411078 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 16837
I20260620 11:35:50.411218 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:35:50.411434 3022 external_mini_cluster.cc:1428] Setting key 95109bd1b6d2abf762c5de52acadc122
Jun 20 11:35:50 dist-test-slave-ctbw krb5kdc[16730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955350, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:35:50.417177 16968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:50.407494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.129:47803)
0620 11:35:50.407630 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:50.407675 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:50.408188 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:50.408505 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:50.408510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:50.409120 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:50.409317 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:50.409322 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.410044 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.410045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:50.411013 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:50.411015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.411642 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.411643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:50.411762 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:50.412332 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:50.412346 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:50.413935 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:50.415852 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:50.415857 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:50.415859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:50.416076 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:50.416296 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:50.416296 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:50.416297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:50.416400 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:50.416694 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:50.416698 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:50.416812 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 11:35:50.416892 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 11:35:50.417009 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
I20260620 11:35:50.417178 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:50.407528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:47803 (local address 127.2.243.190:43247)
0620 11:35:50.407571 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:50.407573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:50.407761 (+ 188us) server_negotiation.cc:407] Connection header received
0620 11:35:50.408337 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:50.408339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:50.408352 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:50.408424 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:50.409437 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.409949 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.411131 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.411470 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.411550 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:50.414027 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:50.414029 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:50.414029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:50.414038 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:50.415764 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:50.416154 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:50.416156 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:50.416156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:50.416223 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:50.416466 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:50.416467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:50.416468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:50.416623 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:50.416704 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:50.416940 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 11:35:50.417040 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:35:50.417788 16966 heartbeater.cc:347] Connected to a master server at 127.2.243.190:43247
I20260620 11:35:50.417900 16966 heartbeater.cc:464] Registering TS with master...
I20260620 11:35:50.418115 16966 heartbeater.cc:511] Master 127.2.243.190:43247 requested a full tablet report, sending...
I20260620 11:35:50.418678 16775 ts_manager.cc:194] Registered new tserver with Master: 7371fe0ea9e04b2196329df942c10b7d (127.2.243.129:39513)
I20260620 11:35:50.419296 16775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:47803
WARNING: no policy specified for kudu/127.2.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
I20260620 11:35:50.437193 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:43247
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:35:50.518078 16971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:50.518313 16971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:50.518352 16971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:50.519526 16971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:50.519593 16971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:50.519636 16971 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:35:50.521112 16971 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:43247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:50.521381 16971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:50.521615 16971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:50.526508 16976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:50.526521 16977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:50.526587 16979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:50.526737 16971 server_base.cc:1034] running on GCE node
I20260620 11:35:50.526993 16971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:50.527184 16971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:50.528344 16971 hybrid_clock.cc:648] HybridClock initialized: now 1781955350528342 us; error 46 us; skew 500 ppm
Jun 20 11:35:50 dist-test-slave-ctbw krb5kdc[16730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955350, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:50.530792 16971 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:35:50.531283 16971 webserver.cc:460] Webserver started at http://127.2.243.130:43403/ using document root <none> and password file <none>
I20260620 11:35:50.531523 16971 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:50.531594 16971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:35:50.531765 16971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:35:50.532807 16971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da"
format_stamp: "Formatted at 2026-06-20 11:35:50 on dist-test-slave-ctbw"
server_key: "0e83df9571a3eb44a5027c9159b067f2"
server_key_iv: "331b2c7630ae01b3695ec30fe6ad4086"
server_key_version: "encryptionkey@0"
I20260620 11:35:50.533164 16971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da"
format_stamp: "Formatted at 2026-06-20 11:35:50 on dist-test-slave-ctbw"
server_key: "0e83df9571a3eb44a5027c9159b067f2"
server_key_iv: "331b2c7630ae01b3695ec30fe6ad4086"
server_key_version: "encryptionkey@0"
I20260620 11:35:50.534498 16971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:35:50.535300 16986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:50.535470 16971 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:50.535589 16971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da"
format_stamp: "Formatted at 2026-06-20 11:35:50 on dist-test-slave-ctbw"
server_key: "0e83df9571a3eb44a5027c9159b067f2"
server_key_iv: "331b2c7630ae01b3695ec30fe6ad4086"
server_key_version: "encryptionkey@0"
I20260620 11:35:50.535667 16971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:50.559969 16971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:50.562260 16971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:50.562438 16971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:50.562700 16971 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:35:50.563079 16971 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:35:50.563135 16971 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:50.563191 16971 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:35:50.563220 16971 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:50.569180 16971 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:36443
I20260620 11:35:50.569213 17099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:36443 every 8 connection(s)
I20260620 11:35:50.569547 16971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:35:50.572248 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 16971
I20260620 11:35:50.572342 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:35:50.572584 3022 external_mini_cluster.cc:1428] Setting key 24a9f5bf5b89c16e8f2856bb739a4dd8
Jun 20 11:35:50 dist-test-slave-ctbw krb5kdc[16730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955350, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:35:50.579748 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:50.570308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:50719 (local address 127.2.243.190:43247)
0620 11:35:50.570371 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:50.570374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:50.570576 (+ 202us) server_negotiation.cc:407] Connection header received
0620 11:35:50.571100 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:50.571101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:50.571112 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:50.571178 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:50.572183 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.572835 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.573952 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.574319 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.574396 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:50.576703 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:50.576704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:50.576705 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:50.576713 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:50.578315 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:50.578735 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:50.578736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:50.578737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:50.578802 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:50.579081 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:50.579081 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:50.579082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:50.579211 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:50.579289 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:50.579566 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 11:35:50.579656 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 11:35:50.579973 17102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:50.570264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.130:50719)
0620 11:35:50.570444 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:50.570488 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:50.571007 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:50.571271 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:50.571275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:50.571837 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:50.572063 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:50.572068 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.572928 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.572929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:50.573864 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:50.573867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.574404 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.574405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:50.574511 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:50.575145 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:50.575154 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:50.576598 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:50.578414 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:50.578421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:50.578423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:50.578659 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:50.578890 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:50.578891 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:50.578892 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:50.579012 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:50.579279 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:50.579286 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:50.579430 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 11:35:50.579531 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 11:35:50.579707 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260620 11:35:50.580468 17100 heartbeater.cc:347] Connected to a master server at 127.2.243.190:43247
I20260620 11:35:50.580552 17100 heartbeater.cc:464] Registering TS with master...
I20260620 11:35:50.580760 17100 heartbeater.cc:511] Master 127.2.243.190:43247 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.2.243.131@KRBTEST.COM; defaulting to no policy
I20260620 11:35:50.581173 16775 ts_manager.cc:194] Registered new tserver with Master: bf36ce71d6e542a4ac6f936f8dc2f9da (127.2.243.130:36443)
I20260620 11:35:50.581548 16775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:50719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
I20260620 11:35:50.597841 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:43247
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:35:50.679198 17105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:50.679448 17105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:50.679487 17105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:50.680671 17105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:50.680742 17105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:50.680788 17105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:35:50.682293 17105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:43247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17105
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:50.682556 17105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:50.682792 17105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:50.687703 17111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:50.687703 17113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:50.687762 17110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:50.688078 17105 server_base.cc:1034] running on GCE node
I20260620 11:35:50.688335 17105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:50.688601 17105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:50.689774 17105 hybrid_clock.cc:648] HybridClock initialized: now 1781955350689757 us; error 34 us; skew 500 ppm
Jun 20 11:35:50 dist-test-slave-ctbw krb5kdc[16730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955350, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:50.692521 17105 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:35:50.693001 17105 webserver.cc:460] Webserver started at http://127.2.243.131:39715/ using document root <none> and password file <none>
I20260620 11:35:50.693248 17105 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:50.693318 17105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:35:50.693468 17105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:35:50.694528 17105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "58e345277f5444f7aa9d70d1411e76b9"
format_stamp: "Formatted at 2026-06-20 11:35:50 on dist-test-slave-ctbw"
server_key: "88135f8d3b3a54b5575e42fd78fdcf09"
server_key_iv: "75a5a00023ccbbf33e76447440610a66"
server_key_version: "encryptionkey@0"
I20260620 11:35:50.694947 17105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "58e345277f5444f7aa9d70d1411e76b9"
format_stamp: "Formatted at 2026-06-20 11:35:50 on dist-test-slave-ctbw"
server_key: "88135f8d3b3a54b5575e42fd78fdcf09"
server_key_iv: "75a5a00023ccbbf33e76447440610a66"
server_key_version: "encryptionkey@0"
I20260620 11:35:50.696338 17105 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:35:50.697134 17120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:50.697471 17105 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:35:50.697593 17105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "58e345277f5444f7aa9d70d1411e76b9"
format_stamp: "Formatted at 2026-06-20 11:35:50 on dist-test-slave-ctbw"
server_key: "88135f8d3b3a54b5575e42fd78fdcf09"
server_key_iv: "75a5a00023ccbbf33e76447440610a66"
server_key_version: "encryptionkey@0"
I20260620 11:35:50.697675 17105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:50.723054 17105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:50.725121 17105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:50.725284 17105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:50.725530 17105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:35:50.725857 17105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:35:50.725914 17105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:50.725960 17105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:35:50.725983 17105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:50.731925 17105 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:40403
I20260620 11:35:50.731964 17233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:40403 every 8 connection(s)
I20260620 11:35:50.732292 17105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:35:50.732748 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 17105
I20260620 11:35:50.732867 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:35:50.733059 3022 external_mini_cluster.cc:1428] Setting key a23975a711107e9f7d7468d752d7e523
Jun 20 11:35:50 dist-test-slave-ctbw krb5kdc[16730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955350, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:35:50.741967 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:50.733016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:33181 (local address 127.2.243.190:43247)
0620 11:35:50.733069 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:50.733071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:50.733361 (+ 290us) server_negotiation.cc:407] Connection header received
0620 11:35:50.733928 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:50.733929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:50.733939 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:50.733979 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:50.735214 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.735725 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.736753 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.736965 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.737052 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:50.739355 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:50.739356 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:50.739356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:50.739363 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:50.740745 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:50.741125 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:50.741126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:50.741127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:50.741183 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:50.741438 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:50.741438 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:50.741439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:50.741544 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:50.741603 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:50.741827 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 11:35:50.741902 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:35:50.742141 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:50.732986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.131:33181)
0620 11:35:50.733233 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:50.733276 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:50.733837 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:50.734065 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:50.734069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:50.734835 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:50.735120 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:50.735125 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.735824 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.735825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:50.736644 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:50.736647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.737059 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.737060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:50.737190 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:50.737658 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:50.737667 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:50.739267 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:50.740815 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:50.740821 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:50.740823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:50.741041 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:50.741275 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:50.741276 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:50.741276 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:50.741379 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:50.741605 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:50.741609 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:50.741722 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 11:35:50.741803 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 11:35:50.741918 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
I20260620 11:35:50.742604 17234 heartbeater.cc:347] Connected to a master server at 127.2.243.190:43247
I20260620 11:35:50.742686 17234 heartbeater.cc:464] Registering TS with master...
I20260620 11:35:50.742929 17234 heartbeater.cc:511] Master 127.2.243.190:43247 requested a full tablet report, sending...
I20260620 11:35:50.743377 16775 ts_manager.cc:194] Registered new tserver with Master: 58e345277f5444f7aa9d70d1411e76b9 (127.2.243.131:40403)
I20260620 11:35:50.743719 16775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:33181
I20260620 11:35:50.745129 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:35:50.752305 16814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:50.746443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.2.243.190:43247)
0620 11:35:50.746481 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:50.746483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:50.746561 (+ 78us) server_negotiation.cc:407] Connection header received
0620 11:35:50.746629 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:50.746629 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:50.746637 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:50.746692 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:50.747292 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.747772 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.748443 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.748631 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.748681 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:50.749822 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:50.749823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:50.749823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:50.749829 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:50.751301 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:50.751615 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:50.751616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:50.751616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:50.751664 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:50.751820 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:50.751821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:50.751821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:50.751948 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:50.752025 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:50.752182 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 11:35:50.752240 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 11:35:50.753856 16775 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:35:50.755628 16775 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:35:50.762728 17248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:50.758195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:40403 (local address 127.0.0.1:50878)
0620 11:35:50.758946 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:50.758957 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:50.759006 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:50.759214 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:50.759215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:50.759218 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:35:50.759274 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:50.759276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.760614 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.760615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:50.761448 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:50.761450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.762396 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.762397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:50.762478 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:50.762494 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 11:35:50.762565 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 11:35:50.762622 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":69,"threads_started":1}
I20260620 11:35:50.762844 17247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:50.758077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44600 (local address 127.2.243.130:36443)
0620 11:35:50.758524 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:50.758528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:50.758540 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:35:50.758584 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:50.758586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:50.758601 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:50.758687 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:35:50.758851 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.760281 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.761929 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.762495 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.762574 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:50.762629 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:50.762709 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 11:35:50.762753 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":103,"threads_started":1}
I20260620 11:35:50.762847 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:50.758289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35822 (local address 127.2.243.129:39513)
0620 11:35:50.758466 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:50.758472 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:50.758904 (+ 432us) server_negotiation.cc:407] Connection header received
0620 11:35:50.759034 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:50.759037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:50.759054 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:50.759108 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:35:50.759402 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.760081 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.761690 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.762365 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.762422 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:50.762471 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:50.762717 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 11:35:50.762763 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":106,"threads_started":1}
I20260620 11:35:50.763062 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:50.757996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:39513 (local address 127.0.0.1:35822)
0620 11:35:50.758802 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:50.758813 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:50.758933 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:50.759269 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:50.759269 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:50.759270 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:35:50.759312 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:50.759314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.760206 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.760207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:50.761592 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:50.761594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.762504 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.762505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:50.762612 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:50.762616 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:35:50.762945 (+ 329us) client_negotiation.cc:240] Negotiation successful
0620 11:35:50.762986 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":790,"mutex_wait_us":100,"thread_start_us":79,"threads_started":1}
I20260620 11:35:50.763140 17249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:50.758293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50878 (local address 127.2.243.131:40403)
0620 11:35:50.758560 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:50.758563 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:50.759059 (+ 496us) server_negotiation.cc:407] Connection header received
0620 11:35:50.759102 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:50.759105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:50.759121 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:50.759198 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:35:50.759419 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.760475 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.761541 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.762290 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.762383 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:50.762449 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:50.762977 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 11:35:50.763045 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":80,"threads_started":1}
I20260620 11:35:50.763662 17246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:50.757980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:36443 (local address 127.0.0.1:44600)
0620 11:35:50.758127 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:50.758172 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:50.758242 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:50.758704 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:50.758707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:50.758709 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:35:50.758765 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:50.758766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.760388 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.760389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:50.761847 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:50.761849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.762579 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.762580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:50.762630 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:50.762633 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:35:50.762694 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 11:35:50.762736 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260620 11:35:50.763918 16900 tablet_service.cc:1467] Processing CreateTablet for tablet cc9f7c89ea2e47cf837d75301cf3edd6 (DEFAULT_TABLE table=test-table [id=e0af47353023498eb8c8146751b107f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:35:50.763918 17168 tablet_service.cc:1467] Processing CreateTablet for tablet cc9f7c89ea2e47cf837d75301cf3edd6 (DEFAULT_TABLE table=test-table [id=e0af47353023498eb8c8146751b107f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:35:50.764295 17168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9f7c89ea2e47cf837d75301cf3edd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:35:50.764295 16900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9f7c89ea2e47cf837d75301cf3edd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:35:50.767447 17034 tablet_service.cc:1467] Processing CreateTablet for tablet cc9f7c89ea2e47cf837d75301cf3edd6 (DEFAULT_TABLE table=test-table [id=e0af47353023498eb8c8146751b107f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:35:50.767766 17252 tablet_bootstrap.cc:492] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9: Bootstrap starting.
I20260620 11:35:50.767791 17034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc9f7c89ea2e47cf837d75301cf3edd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:35:50.768456 17252 tablet_bootstrap.cc:654] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9: Neither blocks nor log segments found. Creating new log.
I20260620 11:35:50.769099 17252 log.cc:824] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9: Log is configured to *not* fsync() on all Append() calls
I20260620 11:35:50.769462 17251 tablet_bootstrap.cc:492] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d: Bootstrap starting.
I20260620 11:35:50.769907 17252 tablet_bootstrap.cc:492] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9: No bootstrap required, opened a new log
I20260620 11:35:50.769991 17252 ts_tablet_manager.cc:1397] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:35:50.770108 17251 tablet_bootstrap.cc:654] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d: Neither blocks nor log segments found. Creating new log.
I20260620 11:35:50.770677 17251 log.cc:824] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d: Log is configured to *not* fsync() on all Append() calls
I20260620 11:35:50.771322 17255 tablet_bootstrap.cc:492] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da: Bootstrap starting.
I20260620 11:35:50.771963 17255 tablet_bootstrap.cc:654] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da: Neither blocks nor log segments found. Creating new log.
I20260620 11:35:50.772012 17252 raft_consensus.cc:348] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } } peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } }
I20260620 11:35:50.772224 17252 raft_consensus.cc:374] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:35:50.772265 17252 raft_consensus.cc:729] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e345277f5444f7aa9d70d1411e76b9, State: Initialized, Role: FOLLOWER
I20260620 11:35:50.772414 17252 consensus_queue.cc:260] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } } peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } }
I20260620 11:35:50.772549 17255 log.cc:824] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da: Log is configured to *not* fsync() on all Append() calls
I20260620 11:35:50.772686 17252 ts_tablet_manager.cc:1428] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:35:50.772698 17234 heartbeater.cc:503] Master 127.2.243.190:43247 was elected leader, sending a full tablet report...
I20260620 11:35:50.773386 17251 tablet_bootstrap.cc:492] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d: No bootstrap required, opened a new log
I20260620 11:35:50.773463 17251 ts_tablet_manager.cc:1397] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260620 11:35:50.773595 17255 tablet_bootstrap.cc:492] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da: No bootstrap required, opened a new log
I20260620 11:35:50.773662 17255 ts_tablet_manager.cc:1397] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:35:50.775652 17255 raft_consensus.cc:348] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } } peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } }
I20260620 11:35:50.775833 17255 raft_consensus.cc:374] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:35:50.775871 17255 raft_consensus.cc:729] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf36ce71d6e542a4ac6f936f8dc2f9da, State: Initialized, Role: FOLLOWER
I20260620 11:35:50.775767 17251 raft_consensus.cc:348] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } } peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } }
I20260620 11:35:50.775944 17251 raft_consensus.cc:374] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:35:50.775996 17255 consensus_queue.cc:260] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } } peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } }
I20260620 11:35:50.775988 17251 raft_consensus.cc:729] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7371fe0ea9e04b2196329df942c10b7d, State: Initialized, Role: FOLLOWER
I20260620 11:35:50.776152 17251 consensus_queue.cc:260] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } } peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } }
I20260620 11:35:50.776280 17255 ts_tablet_manager.cc:1428] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:35:50.776288 17100 heartbeater.cc:503] Master 127.2.243.190:43247 was elected leader, sending a full tablet report...
I20260620 11:35:50.776410 17251 ts_tablet_manager.cc:1428] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 11:35:50.776546 16966 heartbeater.cc:503] Master 127.2.243.190:43247 was elected leader, sending a full tablet report...
W20260620 11:35:50.820190 17101 tablet.cc:2369] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:35:50.831945 17256 raft_consensus.cc:482] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:35:50.832074 17256 raft_consensus.cc:504] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } } peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } }
I20260620 11:35:50.832399 17256 leader_election.cc:288] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf36ce71d6e542a4ac6f936f8dc2f9da (127.2.243.130:36443), 7371fe0ea9e04b2196329df942c10b7d (127.2.243.129:39513)
I20260620 11:35:50.836233 17247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:50.832656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:54527 (local address 127.2.243.130:36443)
0620 11:35:50.832704 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:50.832706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:50.832750 (+ 44us) server_negotiation.cc:407] Connection header received
0620 11:35:50.832779 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:50.832780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:50.832792 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:50.832867 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:35:50.833053 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.833818 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.835282 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.835869 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.835948 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:50.835956 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:50.836126 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 11:35:50.836166 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:35:50.836246 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:50.832705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:53523 (local address 127.2.243.129:39513)
0620 11:35:50.832746 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:50.832748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:50.833061 (+ 313us) server_negotiation.cc:407] Connection header received
0620 11:35:50.833089 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:50.833090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:50.833100 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:50.833151 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:35:50.833305 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.833984 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.835307 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.835869 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.835948 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:50.835956 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:50.836152 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 11:35:50.836189 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:35:50.836246 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:50.832581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:36443 (local address 127.2.243.131:54527)
0620 11:35:50.832645 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:50.832658 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:50.832699 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:50.832890 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:50.832891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:50.832894 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:35:50.832971 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:50.832973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.833893 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.833894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:50.835180 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:50.835184 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.835958 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.835959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:50.836041 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:50.836043 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:35:50.836105 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 11:35:50.836154 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":62}
I20260620 11:35:50.836256 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:50.832594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:39513 (local address 127.2.243.131:53523)
0620 11:35:50.832787 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:50.832794 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:50.832834 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:50.833169 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:50.833169 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:50.833171 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:35:50.833232 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:50.833233 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.834057 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.834058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:50.835231 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:50.835232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.835958 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.835959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:50.836038 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:50.836042 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:35:50.836145 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 11:35:50.836174 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":4,"thread_start_us":109,"threads_started":1}
I20260620 11:35:50.836714 17054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f7c89ea2e47cf837d75301cf3edd6" candidate_uuid: "58e345277f5444f7aa9d70d1411e76b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" is_pre_election: true
I20260620 11:35:50.836905 17054 raft_consensus.cc:2454] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58e345277f5444f7aa9d70d1411e76b9 in term 0.
I20260620 11:35:50.836871 16920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f7c89ea2e47cf837d75301cf3edd6" candidate_uuid: "58e345277f5444f7aa9d70d1411e76b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7371fe0ea9e04b2196329df942c10b7d" is_pre_election: true
I20260620 11:35:50.836993 16920 raft_consensus.cc:2454] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58e345277f5444f7aa9d70d1411e76b9 in term 0.
I20260620 11:35:50.837138 17122 leader_election.cc:302] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58e345277f5444f7aa9d70d1411e76b9, bf36ce71d6e542a4ac6f936f8dc2f9da; no voters:
I20260620 11:35:50.837268 17256 raft_consensus.cc:2781] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:35:50.837363 17256 raft_consensus.cc:482] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:35:50.837406 17256 raft_consensus.cc:3037] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:35:50.838132 17256 raft_consensus.cc:504] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } } peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } }
I20260620 11:35:50.838297 17256 leader_election.cc:288] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9 [CANDIDATE]: Term 1 election: Requested vote from peers bf36ce71d6e542a4ac6f936f8dc2f9da (127.2.243.130:36443), 7371fe0ea9e04b2196329df942c10b7d (127.2.243.129:39513)
I20260620 11:35:50.838444 17054 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f7c89ea2e47cf837d75301cf3edd6" candidate_uuid: "58e345277f5444f7aa9d70d1411e76b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da"
I20260620 11:35:50.838459 16920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc9f7c89ea2e47cf837d75301cf3edd6" candidate_uuid: "58e345277f5444f7aa9d70d1411e76b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7371fe0ea9e04b2196329df942c10b7d"
I20260620 11:35:50.838523 17054 raft_consensus.cc:3037] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:35:50.838546 16920 raft_consensus.cc:3037] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:35:50.839313 17054 raft_consensus.cc:2454] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58e345277f5444f7aa9d70d1411e76b9 in term 1.
I20260620 11:35:50.839489 17122 leader_election.cc:302] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58e345277f5444f7aa9d70d1411e76b9, bf36ce71d6e542a4ac6f936f8dc2f9da; no voters:
I20260620 11:35:50.839530 16920 raft_consensus.cc:2454] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58e345277f5444f7aa9d70d1411e76b9 in term 1.
I20260620 11:35:50.839581 17256 raft_consensus.cc:2781] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:35:50.839762 17256 raft_consensus.cc:686] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9 [term 1 LEADER]: Becoming Leader. State: Replica: 58e345277f5444f7aa9d70d1411e76b9, State: Running, Role: LEADER
I20260620 11:35:50.839877 17256 consensus_queue.cc:237] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } } peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } }
I20260620 11:35:50.840758 16775 catalog_manager.cc:5495] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58e345277f5444f7aa9d70d1411e76b9 (127.2.243.131). New cstate: current_term: 1 leader_uuid: "58e345277f5444f7aa9d70d1411e76b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } health_report { overall_health: UNKNOWN } } }
I20260620 11:35:50.853747 17249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:50.851548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50882 (local address 127.2.243.131:40403)
0620 11:35:50.851603 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:50.851605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:50.851634 (+ 29us) server_negotiation.cc:407] Connection header received
0620 11:35:50.851671 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:50.851672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:50.851687 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:50.851747 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:35:50.851861 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.852339 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.853046 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:50.853225 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:50.853275 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:50.853403 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:35:50.853496 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:35:50.853551 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:50.853590 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 11:35:50.853605 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 11:35:50.855615 17054 raft_consensus.cc:1261] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da [term 1 FOLLOWER]: Refusing update from remote peer 58e345277f5444f7aa9d70d1411e76b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:35:50.855933 16920 raft_consensus.cc:1261] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Refusing update from remote peer 58e345277f5444f7aa9d70d1411e76b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:35:50.855937 17256 consensus_queue.cc:1035] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:35:50.856222 17256 consensus_queue.cc:1035] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:35:50.860733 16775 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:35:50.860855 16775 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:35:50.861572 16775 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55: Sending DeleteTablet for 3 replicas of tablet cc9f7c89ea2e47cf837d75301cf3edd6
I20260620 11:35:50.861902 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 16837
I20260620 11:35:50.861980 16900 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9f7c89ea2e47cf837d75301cf3edd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:35:50 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:35822
I20260620 11:35:50.862498 17034 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9f7c89ea2e47cf837d75301cf3edd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:35:50 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:44600
I20260620 11:35:50.862694 17168 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9f7c89ea2e47cf837d75301cf3edd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:35:50 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:50878
I20260620 11:35:50.862929 17275 tablet_replica.cc:330] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9: stopping tablet replica
I20260620 11:35:50.863054 17275 raft_consensus.cc:2229] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:35:50.863196 17275 raft_consensus.cc:2258] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:35:50.867596 17274 tablet_replica.cc:330] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da: stopping tablet replica
W20260620 11:35:50.867722 16762 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.243.129:39513 (error 108)
I20260620 11:35:50.867797 17275 ts_tablet_manager.cc:1905] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:35:50.867784 17274 raft_consensus.cc:2229] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:35:50.868031 17274 raft_consensus.cc:2258] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:35:50.868292 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 16971
I20260620 11:35:50.869812 17275 ts_tablet_manager.cc:1918] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:35:50.869892 17275 log.cc:1196] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/cc9f7c89ea2e47cf837d75301cf3edd6
I20260620 11:35:50.870193 17275 ts_tablet_manager.cc:1939] T cc9f7c89ea2e47cf837d75301cf3edd6 P 58e345277f5444f7aa9d70d1411e76b9: Deleting consensus metadata
I20260620 11:35:50.870324 17246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:50.870172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:39513 (local address 127.0.0.1:35828)
0620 11:35:50.870244 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:50.870283 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.129:39513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 11:35:50.870537 16762 catalog_manager.cc:4584] TS 7371fe0ea9e04b2196329df942c10b7d (127.2.243.129:39513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cc9f7c89ea2e47cf837d75301cf3edd6: Network error: Client connection negotiation failed: client connection to 127.2.243.129:39513: connect: Connection refused (error 111)
I20260620 11:35:50.870631 16760 catalog_manager.cc:4841] TS 58e345277f5444f7aa9d70d1411e76b9 (127.2.243.131:40403): tablet cc9f7c89ea2e47cf837d75301cf3edd6 (table test-table [id=e0af47353023498eb8c8146751b107f3]) successfully deleted
I20260620 11:35:50.874001 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 17105
I20260620 11:35:50.874032 17246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:50.873925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:36443 (local address 127.0.0.1:44604)
0620 11:35:50.873972 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:50.873998 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.130:36443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 11:35:50.879105 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 16743
I20260620 11:35:50.884367 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:43247
--webserver_interface=127.2.243.190
--webserver_port=46295
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:43247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:35:50.968529 17277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:50.968783 17277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:50.968811 17277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:50.970139 17277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:35:50.970191 17277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:50.970209 17277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:50.970227 17277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:35:50.970292 17277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:35:50.971953 17277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:43247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:43247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=46295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:50.972208 17277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:50.972484 17277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:50.981300 17284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:50.981376 17286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:50.981300 17283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:50.981644 17277 server_base.cc:1034] running on GCE node
I20260620 11:35:50.981925 17277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:50.982249 17277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:50.983413 17277 hybrid_clock.cc:648] HybridClock initialized: now 1781955350983407 us; error 38 us; skew 500 ppm
Jun 20 11:35:50 dist-test-slave-ctbw krb5kdc[16730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955350, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:50.989060 17277 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:35:50.989585 17277 webserver.cc:460] Webserver started at http://127.2.243.190:46295/ using document root <none> and password file <none>
I20260620 11:35:50.989866 17277 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:50.990000 17277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:35:50.991499 17277 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:35:50.992210 17293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:50.992427 17277 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:35:50.992556 17277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "4d3c51a014204e628be7105bd6536a55"
format_stamp: "Formatted at 2026-06-20 11:35:50 on dist-test-slave-ctbw"
server_key: "c80bf89a0991fe682ce46b9d61cd9344"
server_key_iv: "5e2db0a71baa6413d108e0efbbee123f"
server_key_version: "encryptionkey@0"
I20260620 11:35:50.992960 17277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:51.001818 17277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:51.003805 17277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:51.003961 17277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:51.007876 17277 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:43247
I20260620 11:35:51.007911 17345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:43247 every 8 connection(s)
I20260620 11:35:51.008307 17277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:35:51.009225 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 17277
I20260620 11:35:51.009573 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:39513
--local_ip_for_outbound_sockets=127.2.243.129
--tserver_master_addrs=127.2.243.190:43247
--webserver_port=37769
--webserver_interface=127.2.243.129
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:35:51.010457 17346 sys_catalog.cc:263] Verifying existing consensus state
I20260620 11:35:51.011371 17346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55: Bootstrap starting.
I20260620 11:35:51.019574 17346 log.cc:824] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55: Log is configured to *not* fsync() on all Append() calls
I20260620 11:35:51.021629 17346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:35:51.021953 17346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55: Bootstrap complete.
I20260620 11:35:51.024331 17346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c51a014204e628be7105bd6536a55" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43247 } }
I20260620 11:35:51.024561 17346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3c51a014204e628be7105bd6536a55, State: Initialized, Role: FOLLOWER
I20260620 11:35:51.024734 17346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c51a014204e628be7105bd6536a55" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43247 } }
I20260620 11:35:51.024792 17346 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:35:51.024843 17346 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:35:51.024891 17346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 1 FOLLOWER]: Advancing to term 2
I20260620 11:35:51.025879 17346 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c51a014204e628be7105bd6536a55" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43247 } }
I20260620 11:35:51.025987 17346 leader_election.cc:302] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d3c51a014204e628be7105bd6536a55; no voters:
I20260620 11:35:51.026188 17346 leader_election.cc:288] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 11:35:51.026222 17350 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 11:35:51.026332 17350 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 2 LEADER]: Becoming Leader. State: Replica: 4d3c51a014204e628be7105bd6536a55, State: Running, Role: LEADER
I20260620 11:35:51.026472 17346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:35:51.026515 17350 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c51a014204e628be7105bd6536a55" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43247 } }
I20260620 11:35:51.026902 17352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d3c51a014204e628be7105bd6536a55. Latest consensus state: current_term: 2 leader_uuid: "4d3c51a014204e628be7105bd6536a55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c51a014204e628be7105bd6536a55" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43247 } } }
I20260620 11:35:51.026917 17351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4d3c51a014204e628be7105bd6536a55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c51a014204e628be7105bd6536a55" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43247 } } }
I20260620 11:35:51.027109 17352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [sys.catalog]: This master's current role is: LEADER
I20260620 11:35:51.027119 17351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [sys.catalog]: This master's current role is: LEADER
I20260620 11:35:51.027343 17359 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:35:51.028268 17359 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:35:51.028370 17359 catalog_manager.cc:1259] Loaded cluster ID: f76c64f9d6174de8ae97c7b6b315675e
I20260620 11:35:51.028448 17359 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:35:51.028792 17359 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 11:35:51.031580 17359 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 11:35:51.034132 17359 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:35:51.034329 17359 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55: Loaded TSK: 0
I20260620 11:35:51.034538 17359 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 11:35:51.098937 17348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:51.099216 17348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:51.099256 17348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:51.100471 17348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:51.100540 17348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:51.100589 17348 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:35:51.102146 17348 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:39513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=37769
--tserver_master_addrs=127.2.243.190:43247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17348
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:51.102416 17348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:51.102656 17348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:51.107777 17376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:51.107800 17373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:51.107801 17374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:51.108280 17348 server_base.cc:1034] running on GCE node
I20260620 11:35:51.108561 17348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:51.108836 17348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:51.110013 17348 hybrid_clock.cc:648] HybridClock initialized: now 1781955351109975 us; error 56 us; skew 500 ppm
Jun 20 11:35:51 dist-test-slave-ctbw krb5kdc[16730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955351, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:51.112743 17348 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:35:51.113267 17348 webserver.cc:460] Webserver started at http://127.2.243.129:37769/ using document root <none> and password file <none>
I20260620 11:35:51.113541 17348 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:51.113626 17348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:35:51.115059 17348 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:35:51.115778 17383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:51.115979 17348 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:35:51.116077 17348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "7371fe0ea9e04b2196329df942c10b7d"
format_stamp: "Formatted at 2026-06-20 11:35:50 on dist-test-slave-ctbw"
server_key: "bf3ab1fb9cf881dd48eff4788687eb08"
server_key_iv: "0df0a2f77e7002c3977fa7570d3fd7be"
server_key_version: "encryptionkey@0"
I20260620 11:35:51.116544 17348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:51.127668 17348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:51.130190 17348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:51.130338 17348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:51.130586 17348 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:35:51.131155 17390 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 11:35:51.132628 17348 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 11:35:51.132685 17348 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 11:35:51.132721 17348 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 11:35:51.133601 17348 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 11:35:51.133647 17348 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 11:35:51.133740 17390 tablet_bootstrap.cc:492] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d: Bootstrap starting.
I20260620 11:35:51.140908 17348 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:39513
I20260620 11:35:51.141194 17497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:39513 every 8 connection(s)
I20260620 11:35:51.141368 17348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:35:51.145030 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 17348
I20260620 11:35:51.145421 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:36443
--local_ip_for_outbound_sockets=127.2.243.130
--tserver_master_addrs=127.2.243.190:43247
--webserver_port=43403
--webserver_interface=127.2.243.130
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:35:51.145926 17390 log.cc:824] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d: Log is configured to *not* fsync() on all Append() calls
I20260620 11:35:51.148789 17390 tablet_bootstrap.cc:492] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:35:51.149104 17390 tablet_bootstrap.cc:492] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d: Bootstrap complete.
I20260620 11:35:51.150346 17390 ts_tablet_manager.cc:1397] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.009s
Jun 20 11:35:51 dist-test-slave-ctbw krb5kdc[16730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955351, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:35:51.151714 17390 raft_consensus.cc:348] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } } peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } }
I20260620 11:35:51.151983 17390 raft_consensus.cc:729] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7371fe0ea9e04b2196329df942c10b7d, State: Initialized, Role: FOLLOWER
I20260620 11:35:51.152161 17390 consensus_queue.cc:260] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } } peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } }
I20260620 11:35:51.152514 17390 ts_tablet_manager.cc:1428] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260620 11:35:51.157225 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.142131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.129:51861)
0620 11:35:51.142315 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:51.142379 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:51.143235 (+ 856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:51.143839 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:51.143844 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:51.144354 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:51.144687 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:51.144697 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.147356 (+ 2659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.147357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:51.148740 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:51.148743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.149577 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.149578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:51.149736 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:51.150486 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:51.150496 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:51.152378 (+ 1882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:51.155161 (+ 2783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:51.155168 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:51.155170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:51.155509 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:51.155840 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:51.155841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:51.155841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:51.155971 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:51.156563 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:51.156570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:51.156724 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 11:35:51.156836 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 11:35:51.157015 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260620 11:35:51.157810 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.142329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:51861 (local address 127.2.243.190:43247)
0620 11:35:51.142538 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:51.142548 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:51.142590 (+ 42us) server_negotiation.cc:407] Connection header received
0620 11:35:51.143530 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:51.143538 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:51.143602 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:51.143750 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:51.144828 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.146179 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.148845 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.149455 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.149577 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:51.152558 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:51.152610 (+ 52us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:51.152610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:51.152625 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:51.155069 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:51.155622 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:51.155633 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:51.155633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:51.155755 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:51.156074 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:51.156075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:51.156076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:51.156467 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:51.156565 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:51.156958 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 11:35:51.157141 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":100,"threads_started":1}
I20260620 11:35:51.159350 17498 heartbeater.cc:347] Connected to a master server at 127.2.243.190:43247
I20260620 11:35:51.159466 17498 heartbeater.cc:464] Registering TS with master...
I20260620 11:35:51.159672 17498 heartbeater.cc:511] Master 127.2.243.190:43247 requested a full tablet report, sending...
I20260620 11:35:51.160459 17310 ts_manager.cc:194] Registered new tserver with Master: 7371fe0ea9e04b2196329df942c10b7d (127.2.243.129:39513)
I20260620 11:35:51.161615 17310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:51861
I20260620 11:35:51.162771 17498 heartbeater.cc:503] Master 127.2.243.190:43247 was elected leader, sending a full tablet report...
Jun 20 11:35:51 dist-test-slave-ctbw krb5kdc[16730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955350, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for kudu/127.2.243.129@KRBTEST.COM
I20260620 11:35:51.172492 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.161101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:39513 (local address 127.0.0.1:35832)
0620 11:35:51.161949 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:51.161981 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:51.162051 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:51.162299 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:51.162303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:51.162673 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:51.162754 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:51.162756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.164053 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.164054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:51.165636 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:51.165639 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.166405 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.166406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:51.166497 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:51.167164 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:51.167171 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:51.168998 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:51.171194 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:51.171201 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:51.171203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:51.171467 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:51.171697 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:51.171698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:51.171698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:51.171762 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:51.172121 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:51.172127 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:51.172177 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 11:35:51.172258 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 11:35:51.172371 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":108,"threads_started":1}
I20260620 11:35:51.172816 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.161389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35832 (local address 127.2.243.129:39513)
0620 11:35:51.161782 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:51.161789 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:51.162129 (+ 340us) server_negotiation.cc:407] Connection header received
0620 11:35:51.162166 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:51.162169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:51.162184 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:51.162242 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:51.162844 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.163949 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.165768 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.166261 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.166328 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:51.169090 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:51.169099 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:51.169099 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:51.169108 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:51.171113 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:51.171556 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:51.171559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:51.171560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:51.171626 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:51.171827 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:51.171828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:51.171829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:51.172035 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:51.172132 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:51.172324 (+ 192us) server_negotiation.cc:299] Negotiation successful
0620 11:35:51.172453 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":133,"threads_started":1}
I20260620 11:35:51.173427 17432 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9f7c89ea2e47cf837d75301cf3edd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:35:50 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:35832
I20260620 11:35:51.174082 17507 tablet_replica.cc:330] stopping tablet replica
I20260620 11:35:51.174193 17507 raft_consensus.cc:2229] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:35:51.174258 17507 raft_consensus.cc:2258] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:35:51.174611 17507 ts_tablet_manager.cc:1905] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:35:51.176855 17507 ts_tablet_manager.cc:1918] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:35:51.176936 17507 log.cc:1196] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/wal/wals/cc9f7c89ea2e47cf837d75301cf3edd6
I20260620 11:35:51.177153 17507 ts_tablet_manager.cc:1939] T cc9f7c89ea2e47cf837d75301cf3edd6 P 7371fe0ea9e04b2196329df942c10b7d: Deleting consensus metadata
I20260620 11:35:51.177704 17297 catalog_manager.cc:4841] TS 7371fe0ea9e04b2196329df942c10b7d (127.2.243.129:39513): tablet cc9f7c89ea2e47cf837d75301cf3edd6 (table test-table [id=e0af47353023498eb8c8146751b107f3]) successfully deleted
W20260620 11:35:51.249625 17503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:51.249889 17503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:51.249936 17503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:51.251267 17503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:51.251333 17503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:51.251377 17503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:35:51.252854 17503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:36443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=43403
--tserver_master_addrs=127.2.243.190:43247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:51.253122 17503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:51.253351 17503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:51.258457 17515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:51.258430 17513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:51.258422 17512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:51.258697 17503 server_base.cc:1034] running on GCE node
I20260620 11:35:51.258967 17503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:51.259225 17503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:51.260404 17503 hybrid_clock.cc:648] HybridClock initialized: now 1781955351260397 us; error 39 us; skew 500 ppm
Jun 20 11:35:51 dist-test-slave-ctbw krb5kdc[16730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955351, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:51.263150 17503 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:35:51.263710 17503 webserver.cc:460] Webserver started at http://127.2.243.130:43403/ using document root <none> and password file <none>
I20260620 11:35:51.263964 17503 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:51.264031 17503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:35:51.265851 17503 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:35:51.266710 17522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:51.266973 17503 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:35:51.267060 17503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da"
format_stamp: "Formatted at 2026-06-20 11:35:50 on dist-test-slave-ctbw"
server_key: "0e83df9571a3eb44a5027c9159b067f2"
server_key_iv: "331b2c7630ae01b3695ec30fe6ad4086"
server_key_version: "encryptionkey@0"
I20260620 11:35:51.267381 17503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:51.276991 17503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:51.279385 17503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:51.279515 17503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:51.279712 17503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:35:51.280158 17529 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 11:35:51.281713 17503 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 11:35:51.281764 17503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 11:35:51.281790 17503 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 11:35:51.282455 17503 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 11:35:51.282487 17503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 11:35:51.282641 17529 tablet_bootstrap.cc:492] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da: Bootstrap starting.
I20260620 11:35:51.289283 17503 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:36443
I20260620 11:35:51.289402 17636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:36443 every 8 connection(s)
I20260620 11:35:51.289734 17503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:35:51.293573 17529 log.cc:824] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da: Log is configured to *not* fsync() on all Append() calls
I20260620 11:35:51.296345 17529 tablet_bootstrap.cc:492] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:35:51.296708 17529 tablet_bootstrap.cc:492] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da: Bootstrap complete.
Jun 20 11:35:51 dist-test-slave-ctbw krb5kdc[16730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955351, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:35:51.297897 17529 ts_tablet_manager.cc:1397] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260620 11:35:51.298167 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 17503
I20260620 11:35:51.298465 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:40403
--local_ip_for_outbound_sockets=127.2.243.131
--tserver_master_addrs=127.2.243.190:43247
--webserver_port=39715
--webserver_interface=127.2.243.131
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:35:51.299588 17529 raft_consensus.cc:348] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } } peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } }
I20260620 11:35:51.299981 17529 raft_consensus.cc:729] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf36ce71d6e542a4ac6f936f8dc2f9da, State: Initialized, Role: FOLLOWER
I20260620 11:35:51.300132 17529 consensus_queue.cc:260] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } } peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } }
I20260620 11:35:51.300415 17529 ts_tablet_manager.cc:1428] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260620 11:35:51.301256 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.290544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:38753 (local address 127.2.243.190:43247)
0620 11:35:51.290615 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:51.290617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:51.290813 (+ 196us) server_negotiation.cc:407] Connection header received
0620 11:35:51.291695 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:51.291696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:51.291710 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:51.291758 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:51.292755 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.293242 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.294609 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.294990 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.295078 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:51.297837 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:51.297839 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:51.297839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:51.297846 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:51.299624 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:51.300176 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:51.300177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:51.300178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:51.300242 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:51.300523 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:51.300524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:51.300525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:51.300662 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:51.300704 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:51.301108 (+ 404us) server_negotiation.cc:299] Negotiation successful
0620 11:35:51.301191 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 11:35:51.301560 17639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.290442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.130:38753)
0620 11:35:51.290655 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:51.290707 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:51.291578 (+ 871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:51.291830 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:51.291836 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:51.292316 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:51.292649 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:51.292658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.293354 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.293355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:51.294501 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:51.294504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.295108 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.295109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:51.295270 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:51.295860 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:51.295878 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:51.297742 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:51.299723 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:51.299730 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:51.299733 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:51.300077 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:51.300316 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:51.300317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:51.300317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:51.300446 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:51.300753 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:51.300760 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:51.300948 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 11:35:51.301067 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 11:35:51.301247 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":111,"threads_started":1}
I20260620 11:35:51.302016 17637 heartbeater.cc:347] Connected to a master server at 127.2.243.190:43247
I20260620 11:35:51.302124 17637 heartbeater.cc:464] Registering TS with master...
I20260620 11:35:51.302313 17637 heartbeater.cc:511] Master 127.2.243.190:43247 requested a full tablet report, sending...
I20260620 11:35:51.302934 17310 ts_manager.cc:194] Registered new tserver with Master: bf36ce71d6e542a4ac6f936f8dc2f9da (127.2.243.130:36443)
I20260620 11:35:51.303504 17310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:38753
I20260620 11:35:51.304299 17637 heartbeater.cc:503] Master 127.2.243.190:43247 was elected leader, sending a full tablet report...
Jun 20 11:35:51 dist-test-slave-ctbw krb5kdc[16730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955350, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for kudu/127.2.243.130@KRBTEST.COM
I20260620 11:35:51.310995 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.303221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:36443 (local address 127.0.0.1:44608)
0620 11:35:51.303270 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:51.303279 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:51.303314 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:51.303818 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:51.303819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:51.303991 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:51.304034 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:51.304036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.305059 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.305061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:51.305572 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:51.305573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.306056 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.306058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:51.306132 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:51.306584 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:51.306590 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:51.308134 (+ 1544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:51.309920 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:51.309921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:51.309921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:51.310130 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:51.310327 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:51.310328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:51.310329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:51.310391 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:51.310640 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:51.310640 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:51.310686 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 11:35:51.310760 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 11:35:51.310892 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 11:35:51.310999 17643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.303301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44608 (local address 127.2.243.130:36443)
0620 11:35:51.303466 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:51.303471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:51.303482 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:35:51.303521 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:51.303523 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:51.303540 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:51.303682 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:51.304126 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.304978 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.305667 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.305971 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.306047 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:51.308238 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:51.308244 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:51.308245 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:51.308250 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:51.309830 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:51.310210 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:51.310212 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:51.310213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:51.310264 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:51.310450 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:51.310451 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:51.310451 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:51.310579 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:51.310640 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:51.310784 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 11:35:51.310852 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":86,"threads_started":1}
I20260620 11:35:51.311609 17571 tablet_service.cc:1514] Processing DeleteTablet for tablet cc9f7c89ea2e47cf837d75301cf3edd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:35:50 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:44608
I20260620 11:35:51.312112 17644 tablet_replica.cc:330] stopping tablet replica
I20260620 11:35:51.312222 17644 raft_consensus.cc:2229] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:35:51.312290 17644 raft_consensus.cc:2258] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:35:51.312605 17644 ts_tablet_manager.cc:1905] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:35:51.314831 17644 ts_tablet_manager.cc:1918] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:35:51.314954 17644 log.cc:1196] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/wal/wals/cc9f7c89ea2e47cf837d75301cf3edd6
I20260620 11:35:51.315186 17644 ts_tablet_manager.cc:1939] T cc9f7c89ea2e47cf837d75301cf3edd6 P bf36ce71d6e542a4ac6f936f8dc2f9da: Deleting consensus metadata
I20260620 11:35:51.315701 17295 catalog_manager.cc:4841] TS bf36ce71d6e542a4ac6f936f8dc2f9da (127.2.243.130:36443): tablet cc9f7c89ea2e47cf837d75301cf3edd6 (table test-table [id=e0af47353023498eb8c8146751b107f3]) successfully deleted
W20260620 11:35:51.383625 17641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:51.383847 17641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:51.383874 17641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:51.385051 17641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:51.385095 17641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:51.385123 17641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:35:51.386538 17641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:40403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=39715
--tserver_master_addrs=127.2.243.190:43247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:51.386735 17641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:51.386945 17641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:51.391978 17652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:51.392076 17641 server_base.cc:1034] running on GCE node
W20260620 11:35:51.391984 17649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:51.391989 17650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:51.392498 17641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:51.392697 17641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:51.393872 17641 hybrid_clock.cc:648] HybridClock initialized: now 1781955351393862 us; error 27 us; skew 500 ppm
Jun 20 11:35:51 dist-test-slave-ctbw krb5kdc[16730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955351, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:51.396430 17641 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:35:51.396924 17641 webserver.cc:460] Webserver started at http://127.2.243.131:39715/ using document root <none> and password file <none>
I20260620 11:35:51.397185 17641 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:51.397275 17641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:35:51.398571 17641 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:35:51.399200 17659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:51.399369 17641 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:51.399474 17641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "58e345277f5444f7aa9d70d1411e76b9"
format_stamp: "Formatted at 2026-06-20 11:35:50 on dist-test-slave-ctbw"
server_key: "88135f8d3b3a54b5575e42fd78fdcf09"
server_key_iv: "75a5a00023ccbbf33e76447440610a66"
server_key_version: "encryptionkey@0"
I20260620 11:35:51.399837 17641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:51.419391 17641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:51.421365 17641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:51.421533 17641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:51.421769 17641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:35:51.422080 17641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:35:51.422135 17641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:51.422176 17641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:35:51.422204 17641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:51.428164 17641 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:40403
I20260620 11:35:51.428210 17772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:40403 every 8 connection(s)
I20260620 11:35:51.428575 17641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:35:51.434341 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 17641
Jun 20 11:35:51 dist-test-slave-ctbw krb5kdc[16730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955351, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:35:51.438036 17501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.429307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:53587 (local address 127.2.243.190:43247)
0620 11:35:51.429378 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:51.429380 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:51.429560 (+ 180us) server_negotiation.cc:407] Connection header received
0620 11:35:51.430027 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:51.430027 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:51.430038 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:51.430102 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:51.431048 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.431535 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.432530 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.432722 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.432790 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:51.435085 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:51.435087 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:51.435087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:51.435095 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:51.436619 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:51.437058 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:51.437059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:51.437060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:51.437119 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:51.437387 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:51.437388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:51.437388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:51.437502 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:51.437580 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:51.437859 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 11:35:51.437940 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 11:35:51.438284 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.429265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.131:53587)
0620 11:35:51.429421 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:51.429463 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:51.429941 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:51.430158 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:51.430162 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:51.430735 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:51.430963 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:51.430969 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.431616 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.431617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:51.432441 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:51.432443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.432800 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.432801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:51.432904 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:51.433389 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:51.433397 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:51.434972 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:51.436709 (+ 1737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:51.436716 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:51.436719 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:51.436962 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:51.437188 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:51.437189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:51.437190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:51.437317 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:51.437571 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:51.437578 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:51.437726 (+ 148us) client_negotiation.cc:769] Sending connection context
0620 11:35:51.437838 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 11:35:51.438001 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260620 11:35:51.438769 17773 heartbeater.cc:347] Connected to a master server at 127.2.243.190:43247
I20260620 11:35:51.438910 17773 heartbeater.cc:464] Registering TS with master...
I20260620 11:35:51.439107 17773 heartbeater.cc:511] Master 127.2.243.190:43247 requested a full tablet report, sending...
I20260620 11:35:51.439559 17310 ts_manager.cc:194] Registered new tserver with Master: 58e345277f5444f7aa9d70d1411e76b9 (127.2.243.131:40403)
I20260620 11:35:51.440084 17310 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:53587
I20260620 11:35:51.441741 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.434885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60358 (local address 127.2.243.190:43247)
0620 11:35:51.434996 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:51.434999 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:51.435008 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:35:51.435053 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:51.435053 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:51.435071 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:51.435145 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:51.435853 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.436477 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.437215 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.437423 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.437509 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:51.438892 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:51.438893 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:51.438894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:51.438901 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:51.440603 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:51.440965 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:51.440966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:51.440966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:51.441020 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:51.441229 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:51.441230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:51.441230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:51.441345 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:51.441517 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:51.441555 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 11:35:51.441638 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":58,"threads_started":1}
I20260620 11:35:51.442495 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:35:51.445410 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.442913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60372 (local address 127.2.243.190:43247)
0620 11:35:51.442955 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:51.442957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:51.443069 (+ 112us) server_negotiation.cc:407] Connection header received
0620 11:35:51.443152 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:51.443153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:51.443188 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:51.443267 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:35:51.443399 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.443861 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.444608 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.444831 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.444905 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:51.445075 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:35:51.445177 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:35:51.445265 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:51.445315 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 11:35:51.445337 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:35:51.445742 17310 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60372:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:35:51.447582 17310 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:35:51.449679 17432 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5c3b1120e74b20beb1395b44049278 (DEFAULT_TABLE table=test-table [id=4ead3ea1f7af46e5981d956dc1853d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:35:51.449775 17571 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5c3b1120e74b20beb1395b44049278 (DEFAULT_TABLE table=test-table [id=4ead3ea1f7af46e5981d956dc1853d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:35:51.449906 17432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5c3b1120e74b20beb1395b44049278. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:35:51.449966 17571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5c3b1120e74b20beb1395b44049278. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:35:51.451370 17390 tablet_bootstrap.cc:492] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d: Bootstrap starting.
I20260620 11:35:51.451576 17529 tablet_bootstrap.cc:492] T 0a5c3b1120e74b20beb1395b44049278 P bf36ce71d6e542a4ac6f936f8dc2f9da: Bootstrap starting.
I20260620 11:35:51.451691 17390 tablet_bootstrap.cc:654] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d: Neither blocks nor log segments found. Creating new log.
I20260620 11:35:51.451862 17529 tablet_bootstrap.cc:654] T 0a5c3b1120e74b20beb1395b44049278 P bf36ce71d6e542a4ac6f936f8dc2f9da: Neither blocks nor log segments found. Creating new log.
I20260620 11:35:51.452286 17390 tablet_bootstrap.cc:492] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d: No bootstrap required, opened a new log
I20260620 11:35:51.452327 17390 ts_tablet_manager.cc:1397] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:35:51.452672 17529 tablet_bootstrap.cc:492] T 0a5c3b1120e74b20beb1395b44049278 P bf36ce71d6e542a4ac6f936f8dc2f9da: No bootstrap required, opened a new log
I20260620 11:35:51.452663 17390 raft_consensus.cc:348] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } } peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } }
I20260620 11:35:51.452737 17529 ts_tablet_manager.cc:1397] T 0a5c3b1120e74b20beb1395b44049278 P bf36ce71d6e542a4ac6f936f8dc2f9da: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:35:51.452734 17390 raft_consensus.cc:374] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:35:51.452777 17390 raft_consensus.cc:729] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7371fe0ea9e04b2196329df942c10b7d, State: Initialized, Role: FOLLOWER
I20260620 11:35:51.452837 17390 consensus_queue.cc:260] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } } peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } }
I20260620 11:35:51.452941 17390 ts_tablet_manager.cc:1428] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:35:51.452893 17529 raft_consensus.cc:348] T 0a5c3b1120e74b20beb1395b44049278 P bf36ce71d6e542a4ac6f936f8dc2f9da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } } peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } }
I20260620 11:35:51.452977 17529 raft_consensus.cc:374] T 0a5c3b1120e74b20beb1395b44049278 P bf36ce71d6e542a4ac6f936f8dc2f9da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:35:51.453032 17529 raft_consensus.cc:729] T 0a5c3b1120e74b20beb1395b44049278 P bf36ce71d6e542a4ac6f936f8dc2f9da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf36ce71d6e542a4ac6f936f8dc2f9da, State: Initialized, Role: FOLLOWER
I20260620 11:35:51.453087 17529 consensus_queue.cc:260] T 0a5c3b1120e74b20beb1395b44049278 P bf36ce71d6e542a4ac6f936f8dc2f9da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } } peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } }
I20260620 11:35:51.453177 17529 ts_tablet_manager.cc:1428] T 0a5c3b1120e74b20beb1395b44049278 P bf36ce71d6e542a4ac6f936f8dc2f9da: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 11:35:51.453670 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.449443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:40403 (local address 127.0.0.1:50886)
0620 11:35:51.449478 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:51.449489 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:51.449525 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:51.449920 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:51.449921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:51.449923 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:35:51.449966 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:51.449968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.451282 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.451283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:51.452425 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:51.452428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.453446 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.453446 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:51.453514 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:51.453541 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 11:35:51.453588 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 11:35:51.453621 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 11:35:51.453711 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.449530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50886 (local address 127.2.243.131:40403)
0620 11:35:51.449726 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:51.449731 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:51.449742 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:35:51.449782 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:51.449784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:51.449801 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:51.450043 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:35:51.450120 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.451164 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.452539 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.453319 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.453392 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:51.453455 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:51.453614 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 11:35:51.453649 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":117,"threads_started":1}
I20260620 11:35:51.454512 17707 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5c3b1120e74b20beb1395b44049278 (DEFAULT_TABLE table=test-table [id=4ead3ea1f7af46e5981d956dc1853d3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:35:51.454859 17707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5c3b1120e74b20beb1395b44049278. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:35:51.457667 17782 tablet_bootstrap.cc:492] T 0a5c3b1120e74b20beb1395b44049278 P 58e345277f5444f7aa9d70d1411e76b9: Bootstrap starting.
I20260620 11:35:51.458242 17782 tablet_bootstrap.cc:654] T 0a5c3b1120e74b20beb1395b44049278 P 58e345277f5444f7aa9d70d1411e76b9: Neither blocks nor log segments found. Creating new log.
I20260620 11:35:51.458691 17782 log.cc:824] T 0a5c3b1120e74b20beb1395b44049278 P 58e345277f5444f7aa9d70d1411e76b9: Log is configured to *not* fsync() on all Append() calls
I20260620 11:35:51.459520 17782 tablet_bootstrap.cc:492] T 0a5c3b1120e74b20beb1395b44049278 P 58e345277f5444f7aa9d70d1411e76b9: No bootstrap required, opened a new log
I20260620 11:35:51.459619 17782 ts_tablet_manager.cc:1397] T 0a5c3b1120e74b20beb1395b44049278 P 58e345277f5444f7aa9d70d1411e76b9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:35:51.460984 17782 raft_consensus.cc:348] T 0a5c3b1120e74b20beb1395b44049278 P 58e345277f5444f7aa9d70d1411e76b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } } peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } }
I20260620 11:35:51.461153 17782 raft_consensus.cc:374] T 0a5c3b1120e74b20beb1395b44049278 P 58e345277f5444f7aa9d70d1411e76b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:35:51.461198 17782 raft_consensus.cc:729] T 0a5c3b1120e74b20beb1395b44049278 P 58e345277f5444f7aa9d70d1411e76b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58e345277f5444f7aa9d70d1411e76b9, State: Initialized, Role: FOLLOWER
I20260620 11:35:51.461329 17782 consensus_queue.cc:260] T 0a5c3b1120e74b20beb1395b44049278 P 58e345277f5444f7aa9d70d1411e76b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } } peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } }
I20260620 11:35:51.461630 17782 ts_tablet_manager.cc:1428] T 0a5c3b1120e74b20beb1395b44049278 P 58e345277f5444f7aa9d70d1411e76b9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:35:51.461733 17773 heartbeater.cc:503] Master 127.2.243.190:43247 was elected leader, sending a full tablet report...
I20260620 11:35:51.465577 17504 raft_consensus.cc:482] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:35:51.465725 17504 raft_consensus.cc:504] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } } peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } }
I20260620 11:35:51.466053 17504 leader_election.cc:288] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58e345277f5444f7aa9d70d1411e76b9 (127.2.243.131:40403), bf36ce71d6e542a4ac6f936f8dc2f9da (127.2.243.130:36443)
I20260620 11:35:51.469389 17643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.466481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:40227 (local address 127.2.243.130:36443)
0620 11:35:51.466518 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:51.466520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:51.466667 (+ 147us) server_negotiation.cc:407] Connection header received
0620 11:35:51.466738 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:51.466739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:51.466763 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:51.466829 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:35:51.467010 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.467628 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.468586 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.469001 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.469101 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:51.469181 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:51.469265 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 11:35:51.469312 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:35:51.469419 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.466349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:42013 (local address 127.2.243.131:40403)
0620 11:35:51.466395 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:51.466397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:51.466407 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:35:51.466535 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:51.466537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:51.466548 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:51.466615 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:35:51.466793 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.467529 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.468601 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.468999 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.469103 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:51.469113 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:51.469287 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 11:35:51.469325 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 11:35:51.469440 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.466276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:40403 (local address 127.2.243.129:42013)
0620 11:35:51.466340 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:51.466352 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:51.466418 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:51.466623 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:51.466624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:51.466626 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:35:51.466711 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:51.466714 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.467618 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.467619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:51.468481 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:51.468484 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.469102 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.469103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:51.469193 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:51.469196 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:35:51.469279 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 11:35:51.469322 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":18}
I20260620 11:35:51.469453 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.466425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:36443 (local address 127.2.243.129:40227)
0620 11:35:51.466597 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:51.466606 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:51.466684 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:51.466836 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:51.466837 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:51.466838 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:35:51.466934 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:51.466936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.467715 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.467716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:51.468487 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:51.468489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.469104 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.469105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:51.469169 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:51.469197 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 11:35:51.469258 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 11:35:51.469300 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":113,"threads_started":1}
I20260620 11:35:51.469871 17727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5c3b1120e74b20beb1395b44049278" candidate_uuid: "7371fe0ea9e04b2196329df942c10b7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e345277f5444f7aa9d70d1411e76b9" is_pre_election: true
I20260620 11:35:51.469923 17591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5c3b1120e74b20beb1395b44049278" candidate_uuid: "7371fe0ea9e04b2196329df942c10b7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" is_pre_election: true
I20260620 11:35:51.470042 17727 raft_consensus.cc:2454] T 0a5c3b1120e74b20beb1395b44049278 P 58e345277f5444f7aa9d70d1411e76b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7371fe0ea9e04b2196329df942c10b7d in term 0.
I20260620 11:35:51.470093 17591 raft_consensus.cc:2454] T 0a5c3b1120e74b20beb1395b44049278 P bf36ce71d6e542a4ac6f936f8dc2f9da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7371fe0ea9e04b2196329df942c10b7d in term 0.
I20260620 11:35:51.470301 17385 leader_election.cc:302] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58e345277f5444f7aa9d70d1411e76b9, 7371fe0ea9e04b2196329df942c10b7d; no voters:
I20260620 11:35:51.470448 17504 raft_consensus.cc:2781] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:35:51.470510 17504 raft_consensus.cc:482] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:35:51.470548 17504 raft_consensus.cc:3037] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:35:51.471405 17504 raft_consensus.cc:504] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } } peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } }
I20260620 11:35:51.471557 17504 leader_election.cc:288] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 1 election: Requested vote from peers 58e345277f5444f7aa9d70d1411e76b9 (127.2.243.131:40403), bf36ce71d6e542a4ac6f936f8dc2f9da (127.2.243.130:36443)
I20260620 11:35:51.471755 17591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5c3b1120e74b20beb1395b44049278" candidate_uuid: "7371fe0ea9e04b2196329df942c10b7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da"
I20260620 11:35:51.471776 17727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5c3b1120e74b20beb1395b44049278" candidate_uuid: "7371fe0ea9e04b2196329df942c10b7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58e345277f5444f7aa9d70d1411e76b9"
I20260620 11:35:51.471848 17727 raft_consensus.cc:3037] T 0a5c3b1120e74b20beb1395b44049278 P 58e345277f5444f7aa9d70d1411e76b9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:35:51.471849 17591 raft_consensus.cc:3037] T 0a5c3b1120e74b20beb1395b44049278 P bf36ce71d6e542a4ac6f936f8dc2f9da [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:35:51.472692 17591 raft_consensus.cc:2454] T 0a5c3b1120e74b20beb1395b44049278 P bf36ce71d6e542a4ac6f936f8dc2f9da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7371fe0ea9e04b2196329df942c10b7d in term 1.
I20260620 11:35:51.472798 17727 raft_consensus.cc:2454] T 0a5c3b1120e74b20beb1395b44049278 P 58e345277f5444f7aa9d70d1411e76b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7371fe0ea9e04b2196329df942c10b7d in term 1.
I20260620 11:35:51.472843 17385 leader_election.cc:302] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7371fe0ea9e04b2196329df942c10b7d, bf36ce71d6e542a4ac6f936f8dc2f9da; no voters:
I20260620 11:35:51.472949 17504 raft_consensus.cc:2781] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:35:51.473127 17504 raft_consensus.cc:686] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 1 LEADER]: Becoming Leader. State: Replica: 7371fe0ea9e04b2196329df942c10b7d, State: Running, Role: LEADER
I20260620 11:35:51.473227 17504 consensus_queue.cc:237] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } } peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } }
I20260620 11:35:51.474035 17309 catalog_manager.cc:5495] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 7371fe0ea9e04b2196329df942c10b7d (127.2.243.129). New cstate: current_term: 1 leader_uuid: "7371fe0ea9e04b2196329df942c10b7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } health_report { overall_health: UNKNOWN } } }
I20260620 11:35:51.483470 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.481026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35848 (local address 127.2.243.129:39513)
0620 11:35:51.481087 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:51.481090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:51.481110 (+ 20us) server_negotiation.cc:407] Connection header received
0620 11:35:51.481144 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:51.481144 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:51.481163 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:51.481244 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:35:51.481378 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.481883 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.482599 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:51.482855 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:51.482947 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:51.483119 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:35:51.483228 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:35:51.483314 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:51.483366 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 11:35:51.483388 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 11:35:51.484973 17727 raft_consensus.cc:1261] T 0a5c3b1120e74b20beb1395b44049278 P 58e345277f5444f7aa9d70d1411e76b9 [term 1 FOLLOWER]: Refusing update from remote peer 7371fe0ea9e04b2196329df942c10b7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:35:51.485018 17591 raft_consensus.cc:1261] T 0a5c3b1120e74b20beb1395b44049278 P bf36ce71d6e542a4ac6f936f8dc2f9da [term 1 FOLLOWER]: Refusing update from remote peer 7371fe0ea9e04b2196329df942c10b7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:35:51.485370 17786 consensus_queue.cc:1035] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:35:51.485536 17504 consensus_queue.cc:1035] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:35:51.487371 17789 mvcc.cc:205] Tried to move back new op lower bound from 7298889119680544768 to 7298889119635652608. Current Snapshot: MvccSnapshot[applied={T|T < 7298889119680544768}]
I20260620 11:35:51.488112 17791 mvcc.cc:205] Tried to move back new op lower bound from 7298889119680544768 to 7298889119635652608. Current Snapshot: MvccSnapshot[applied={T|T < 7298889119680544768}]
I20260620 11:35:51.488229 17790 mvcc.cc:205] Tried to move back new op lower bound from 7298889119680544768 to 7298889119635652608. Current Snapshot: MvccSnapshot[applied={T|T < 7298889119680544768}]
I20260620 11:35:51.489780 17310 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:35:51.489872 17310 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60372:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:35:51.490559 17310 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55: Sending DeleteTablet for 3 replicas of tablet 0a5c3b1120e74b20beb1395b44049278
I20260620 11:35:51.490921 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 17348
I20260620 11:35:51.490911 17707 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5c3b1120e74b20beb1395b44049278 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:35:51 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:50886
I20260620 11:35:51.491007 17571 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5c3b1120e74b20beb1395b44049278 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:35:51 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:44608
I20260620 11:35:51.491092 17432 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5c3b1120e74b20beb1395b44049278 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:35:51 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:35832
I20260620 11:35:51.491169 17507 tablet_replica.cc:330] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d: stopping tablet replica
I20260620 11:35:51.491215 17507 raft_consensus.cc:2229] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:35:51.491297 17507 raft_consensus.cc:2258] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:35:51.491683 17798 tablet_replica.cc:330] T 0a5c3b1120e74b20beb1395b44049278 P 58e345277f5444f7aa9d70d1411e76b9: stopping tablet replica
I20260620 11:35:51.491762 17798 raft_consensus.cc:2229] T 0a5c3b1120e74b20beb1395b44049278 P 58e345277f5444f7aa9d70d1411e76b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:35:51.491802 17644 tablet_replica.cc:330] T 0a5c3b1120e74b20beb1395b44049278 P bf36ce71d6e542a4ac6f936f8dc2f9da: stopping tablet replica
I20260620 11:35:51.491870 17644 raft_consensus.cc:2229] T 0a5c3b1120e74b20beb1395b44049278 P bf36ce71d6e542a4ac6f936f8dc2f9da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:35:51.491887 17798 raft_consensus.cc:2258] T 0a5c3b1120e74b20beb1395b44049278 P 58e345277f5444f7aa9d70d1411e76b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:35:51.491921 17644 raft_consensus.cc:2258] T 0a5c3b1120e74b20beb1395b44049278 P bf36ce71d6e542a4ac6f936f8dc2f9da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:35:51.492254 17644 ts_tablet_manager.cc:1905] T 0a5c3b1120e74b20beb1395b44049278 P bf36ce71d6e542a4ac6f936f8dc2f9da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:35:51.492416 17798 ts_tablet_manager.cc:1905] T 0a5c3b1120e74b20beb1395b44049278 P 58e345277f5444f7aa9d70d1411e76b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:35:51.493829 17798 ts_tablet_manager.cc:1918] T 0a5c3b1120e74b20beb1395b44049278 P 58e345277f5444f7aa9d70d1411e76b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:35:51.493928 17798 log.cc:1196] T 0a5c3b1120e74b20beb1395b44049278 P 58e345277f5444f7aa9d70d1411e76b9: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/0a5c3b1120e74b20beb1395b44049278
I20260620 11:35:51.494048 17644 ts_tablet_manager.cc:1918] T 0a5c3b1120e74b20beb1395b44049278 P bf36ce71d6e542a4ac6f936f8dc2f9da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:35:51.494112 17644 log.cc:1196] T 0a5c3b1120e74b20beb1395b44049278 P bf36ce71d6e542a4ac6f936f8dc2f9da: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/wal/wals/0a5c3b1120e74b20beb1395b44049278
I20260620 11:35:51.494246 17798 ts_tablet_manager.cc:1939] T 0a5c3b1120e74b20beb1395b44049278 P 58e345277f5444f7aa9d70d1411e76b9: Deleting consensus metadata
I20260620 11:35:51.494333 17644 ts_tablet_manager.cc:1939] T 0a5c3b1120e74b20beb1395b44049278 P bf36ce71d6e542a4ac6f936f8dc2f9da: Deleting consensus metadata
I20260620 11:35:51.494693 17295 catalog_manager.cc:4841] TS bf36ce71d6e542a4ac6f936f8dc2f9da (127.2.243.130:36443): tablet 0a5c3b1120e74b20beb1395b44049278 (table test-table [id=4ead3ea1f7af46e5981d956dc1853d3d]) successfully deleted
I20260620 11:35:51.494882 17295 catalog_manager.cc:4841] TS 58e345277f5444f7aa9d70d1411e76b9 (127.2.243.131:40403): tablet 0a5c3b1120e74b20beb1395b44049278 (table test-table [id=4ead3ea1f7af46e5981d956dc1853d3d]) successfully deleted
W20260620 11:35:51.497861 17297 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.243.129:39513 (error 108)
I20260620 11:35:51.498142 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 17503
I20260620 11:35:51.499339 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.498198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:39513 (local address 127.0.0.1:35852)
0620 11:35:51.499268 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:51.499310 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.129:39513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1049}
W20260620 11:35:51.499581 17297 catalog_manager.cc:4584] TS 7371fe0ea9e04b2196329df942c10b7d (127.2.243.129:39513): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a5c3b1120e74b20beb1395b44049278: Network error: Client connection negotiation failed: client connection to 127.2.243.129:39513: connect: Connection refused (error 111)
I20260620 11:35:51.503187 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 17641
I20260620 11:35:51.507638 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 17277
I20260620 11:35:51.513135 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:43247
--webserver_interface=127.2.243.190
--webserver_port=46295
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:43247
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:35:51.597761 17800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:51.597978 17800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:51.598001 17800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:51.599203 17800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:35:51.599244 17800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:51.599259 17800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:51.599300 17800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:35:51.599328 17800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:35:51.600935 17800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:43247
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:43247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=46295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:51.601186 17800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:51.601421 17800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:51.606180 17805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:51.606240 17806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:51.606474 17800 server_base.cc:1034] running on GCE node
W20260620 11:35:51.606194 17808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:51.606819 17800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:51.607126 17800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:51.608268 17800 hybrid_clock.cc:648] HybridClock initialized: now 1781955351608258 us; error 27 us; skew 500 ppm
Jun 20 11:35:51 dist-test-slave-ctbw krb5kdc[16730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955351, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:51.610713 17800 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:35:51.611271 17800 webserver.cc:460] Webserver started at http://127.2.243.190:46295/ using document root <none> and password file <none>
I20260620 11:35:51.611500 17800 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:51.611567 17800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:35:51.612927 17800 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:35:51.613593 17815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:51.613770 17800 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:35:51.613873 17800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "4d3c51a014204e628be7105bd6536a55"
format_stamp: "Formatted at 2026-06-20 11:35:50 on dist-test-slave-ctbw"
server_key: "c80bf89a0991fe682ce46b9d61cd9344"
server_key_iv: "5e2db0a71baa6413d108e0efbbee123f"
server_key_version: "encryptionkey@0"
I20260620 11:35:51.614213 17800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:51.634030 17800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:51.636127 17800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:51.636296 17800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:51.641239 17800 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:43247
I20260620 11:35:51.641269 17867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:43247 every 8 connection(s)
I20260620 11:35:51.641731 17800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:35:51.642279 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 17800
I20260620 11:35:51.642637 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:39513
--local_ip_for_outbound_sockets=127.2.243.129
--tserver_master_addrs=127.2.243.190:43247
--webserver_port=37769
--webserver_interface=127.2.243.129
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:35:51.644164 17868 sys_catalog.cc:263] Verifying existing consensus state
I20260620 11:35:51.644819 17868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55: Bootstrap starting.
I20260620 11:35:51.656097 17868 log.cc:824] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55: Log is configured to *not* fsync() on all Append() calls
I20260620 11:35:51.659232 17868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:35:51.659578 17868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55: Bootstrap complete.
I20260620 11:35:51.662914 17868 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c51a014204e628be7105bd6536a55" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43247 } }
I20260620 11:35:51.663276 17868 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3c51a014204e628be7105bd6536a55, State: Initialized, Role: FOLLOWER
I20260620 11:35:51.663460 17868 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c51a014204e628be7105bd6536a55" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43247 } }
I20260620 11:35:51.663600 17868 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:35:51.663647 17868 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:35:51.663704 17868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 2 FOLLOWER]: Advancing to term 3
I20260620 11:35:51.665081 17868 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c51a014204e628be7105bd6536a55" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43247 } }
I20260620 11:35:51.665201 17868 leader_election.cc:302] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d3c51a014204e628be7105bd6536a55; no voters:
I20260620 11:35:51.665364 17868 leader_election.cc:288] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 11:35:51.665452 17872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 11:35:51.665608 17868 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:35:51.665621 17872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [term 3 LEADER]: Becoming Leader. State: Replica: 4d3c51a014204e628be7105bd6536a55, State: Running, Role: LEADER
I20260620 11:35:51.665836 17872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c51a014204e628be7105bd6536a55" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43247 } }
I20260620 11:35:51.666085 17872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d3c51a014204e628be7105bd6536a55. Latest consensus state: current_term: 3 leader_uuid: "4d3c51a014204e628be7105bd6536a55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c51a014204e628be7105bd6536a55" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43247 } } }
I20260620 11:35:51.666177 17872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [sys.catalog]: This master's current role is: LEADER
I20260620 11:35:51.666131 17873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4d3c51a014204e628be7105bd6536a55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d3c51a014204e628be7105bd6536a55" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 43247 } } }
I20260620 11:35:51.666334 17873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d3c51a014204e628be7105bd6536a55 [sys.catalog]: This master's current role is: LEADER
I20260620 11:35:51.666393 17880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:35:51.667311 17880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:35:51.667486 17880 catalog_manager.cc:1259] Loaded cluster ID: f76c64f9d6174de8ae97c7b6b315675e
I20260620 11:35:51.667711 17880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 11:35:51.667977 17880 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9b44ccc9ad google::LogMessage::Fail() at ??:0
@ 0x7f9b44ccf88a google::LogMessage::SendToLog() at ??:0
@ 0x7f9b44ccc4de google::LogMessage::Flush() at ??:0
@ 0x7f9b44ccda19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9b4cbed0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f9b4cbf68b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9b46b9a0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9b46b8f954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9b479f36db start_thread at ??:0
@ 0x7f9b437ca61f clone at ??:0
W20260620 11:35:51.763243 17870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:51.763455 17870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:51.763481 17870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:51.764729 17870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:51.764771 17870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:51.764799 17870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:35:51.766278 17870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:39513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=37769
--tserver_master_addrs=127.2.243.190:43247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:51.766517 17870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:51.766763 17870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:51.771997 17907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:51.772002 17904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:51.772027 17905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:51.772399 17870 server_base.cc:1034] running on GCE node
I20260620 11:35:51.772600 17870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:51.772862 17870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:51.774027 17870 hybrid_clock.cc:648] HybridClock initialized: now 1781955351774009 us; error 33 us; skew 500 ppm
Jun 20 11:35:51 dist-test-slave-ctbw krb5kdc[16730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955351, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:51.776968 17870 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:35:51.777536 17870 webserver.cc:460] Webserver started at http://127.2.243.129:37769/ using document root <none> and password file <none>
I20260620 11:35:51.777786 17870 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:51.777858 17870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:35:51.779597 17870 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 11:35:51.780328 17914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:51.780619 17870 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:35:51.780733 17870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "7371fe0ea9e04b2196329df942c10b7d"
format_stamp: "Formatted at 2026-06-20 11:35:50 on dist-test-slave-ctbw"
server_key: "bf3ab1fb9cf881dd48eff4788687eb08"
server_key_iv: "0df0a2f77e7002c3977fa7570d3fd7be"
server_key_version: "encryptionkey@0"
I20260620 11:35:51.781191 17870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:51.795177 17870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:51.797824 17870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:51.797988 17870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:51.798269 17870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:35:51.798802 17921 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 11:35:51.800390 17870 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 11:35:51.800450 17870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 11:35:51.800483 17870 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 11:35:51.801319 17870 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 11:35:51.801362 17870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 11:35:51.801440 17921 tablet_bootstrap.cc:492] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d: Bootstrap starting.
I20260620 11:35:51.808514 17870 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:39513
I20260620 11:35:51.808743 18028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:39513 every 8 connection(s)
I20260620 11:35:51.808959 17870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:35:51.809620 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 17870
I20260620 11:35:51.809942 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:36443
--local_ip_for_outbound_sockets=127.2.243.130
--tserver_master_addrs=127.2.243.190:43247
--webserver_port=43403
--webserver_interface=127.2.243.130
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:35:51.813112 17921 log.cc:824] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d: Log is configured to *not* fsync() on all Append() calls
I20260620 11:35:51.815424 17921 tablet_bootstrap.cc:492] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:35:51.815719 17921 tablet_bootstrap.cc:492] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d: Bootstrap complete.
I20260620 11:35:51.816638 17921 ts_tablet_manager.cc:1397] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260620 11:35:51.817641 17921 raft_consensus.cc:348] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } } peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } }
I20260620 11:35:51.817893 17921 raft_consensus.cc:729] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7371fe0ea9e04b2196329df942c10b7d, State: Initialized, Role: FOLLOWER
I20260620 11:35:51.818064 17921 consensus_queue.cc:260] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } } peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } }
I20260620 11:35:51.818351 17921 ts_tablet_manager.cc:1428] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
W20260620 11:35:51.891332 18031 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:35:51.809799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.129:36359)
0620 11:35:51.811067 (+ 1268us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:51.811121 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:51.811961 (+ 840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:51.890949 (+ 78988us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1215,"thread_start_us":104,"threads_started":1}
W20260620 11:35:51.891598 18029 heartbeater.cc:644] Failed to heartbeat to 127.2.243.190:43247 (0 consecutive failures): Network error: Failed to ping master at 127.2.243.190:43247: Client connection negotiation failed: client connection to 127.2.243.190:43247: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 11:35:51.891984 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.891804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.129:53129)
0620 11:35:51.891866 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:51.891958 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 11:35:51.892395 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.892291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.129:60123)
0620 11:35:51.892351 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:51.892374 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260620 11:35:51.892498 18029 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 11:35:51.906011 18032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:51.906219 18032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:51.906242 18032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:51.907444 18032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:51.907485 18032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:51.907512 18032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:35:51.909080 18032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:36443
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=43403
--tserver_master_addrs=127.2.243.190:43247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:51.909271 18032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:51.909438 18032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:51.914355 18042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:51.914326 18040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:51.914392 18032 server_base.cc:1034] running on GCE node
W20260620 11:35:51.914333 18039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:51.914907 18032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:51.915187 18032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:51.916333 18032 hybrid_clock.cc:648] HybridClock initialized: now 1781955351916324 us; error 28 us; skew 500 ppm
Jun 20 11:35:51 dist-test-slave-ctbw krb5kdc[16730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955351, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:51.918745 18032 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:35:51.919198 18032 webserver.cc:460] Webserver started at http://127.2.243.130:43403/ using document root <none> and password file <none>
I20260620 11:35:51.919420 18032 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:51.919466 18032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:35:51.920763 18032 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:35:51.921358 18049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:51.921525 18032 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:35:51.921620 18032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da"
format_stamp: "Formatted at 2026-06-20 11:35:50 on dist-test-slave-ctbw"
server_key: "0e83df9571a3eb44a5027c9159b067f2"
server_key_iv: "331b2c7630ae01b3695ec30fe6ad4086"
server_key_version: "encryptionkey@0"
I20260620 11:35:51.921937 18032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:51.955612 18032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:51.957492 18032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:51.957660 18032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:51.957897 18032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:35:51.958222 18032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:35:51.958279 18032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:51.958323 18032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:35:51.958348 18032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:51.964367 18032 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:36443
I20260620 11:35:51.964408 18162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:36443 every 8 connection(s)
I20260620 11:35:51.964751 18032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:35:51.965802 18165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.965434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.130:35451)
0620 11:35:51.965595 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:51.965667 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
W20260620 11:35:51.966223 18163 heartbeater.cc:644] Failed to heartbeat to 127.2.243.190:43247 (0 consecutive failures): Network error: Failed to ping master at 127.2.243.190:43247: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
I20260620 11:35:51.966588 18165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.966444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.130:44679)
0620 11:35:51.966490 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:51.966549 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 11:35:51.966624 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 18032
I20260620 11:35:51.966969 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:40403
--local_ip_for_outbound_sockets=127.2.243.131
--tserver_master_addrs=127.2.243.190:43247
--webserver_port=39715
--webserver_interface=127.2.243.131
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:40455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:35:51.967139 18165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:51.966993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.130:54501)
0620 11:35:51.967072 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:51.967107 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 11:35:51.967252 18163 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 11:35:52.049396 18166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:35:52.049602 18166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:35:52.049624 18166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:35:52.050766 18166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:35:52.050806 18166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:35:52.050832 18166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:35:52.052285 18166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:40455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:40403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=39715
--tserver_master_addrs=127.2.243.190:43247
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:35:52.052533 18166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:35:52.052757 18166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:35:52.057690 18174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:52.057718 18172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:35:52.057706 18171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:35:52.057802 18166 server_base.cc:1034] running on GCE node
I20260620 11:35:52.058179 18166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:35:52.058368 18166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:35:52.059545 18166 hybrid_clock.cc:648] HybridClock initialized: now 1781955352059532 us; error 45 us; skew 500 ppm
Jun 20 11:35:52 dist-test-slave-ctbw krb5kdc[16730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955352, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:35:52.062199 18166 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:35:52.062647 18166 webserver.cc:460] Webserver started at http://127.2.243.131:39715/ using document root <none> and password file <none>
I20260620 11:35:52.062834 18166 fs_manager.cc:359] Metadata directory not provided
I20260620 11:35:52.062932 18166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:35:52.064265 18166 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:35:52.064831 18181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:52.065048 18166 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:35:52.065120 18166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "58e345277f5444f7aa9d70d1411e76b9"
format_stamp: "Formatted at 2026-06-20 11:35:50 on dist-test-slave-ctbw"
server_key: "88135f8d3b3a54b5575e42fd78fdcf09"
server_key_iv: "75a5a00023ccbbf33e76447440610a66"
server_key_version: "encryptionkey@0"
I20260620 11:35:52.065426 18166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:35:52.079737 18166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:35:52.081544 18166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:35:52.081655 18166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:35:52.081830 18166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:35:52.082146 18166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:35:52.082185 18166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:52.082207 18166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:35:52.082253 18166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:35:52.088634 18166 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:40403
I20260620 11:35:52.088683 18294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:40403 every 8 connection(s)
I20260620 11:35:52.089059 18166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:35:52.090112 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:52.089734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.131:34481)
0620 11:35:52.089888 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:52.089963 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
W20260620 11:35:52.090523 18295 heartbeater.cc:644] Failed to heartbeat to 127.2.243.190:43247 (0 consecutive failures): Network error: Failed to ping master at 127.2.243.190:43247: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
I20260620 11:35:52.090804 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:52.090710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.131:60377)
0620 11:35:52.090756 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:52.090786 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 11:35:52.091109 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:52.091053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.131:53073)
0620 11:35:52.091075 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:52.091094 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260620 11:35:52.091223 18295 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 11:35:52.092181 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 18166
I20260620 11:35:52.893127 18299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:52.892864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.129:39709)
0620 11:35:52.892997 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:52.893041 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260620 11:35:52.967891 18300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:52.967587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.130:57307)
0620 11:35:52.967777 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:52.967831 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":119,"threads_started":1}
I20260620 11:35:53.074587 18301 raft_consensus.cc:482] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:35:53.074715 18301 raft_consensus.cc:504] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } } peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } }
I20260620 11:35:53.075129 18301 leader_election.cc:288] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58e345277f5444f7aa9d70d1411e76b9 (127.2.243.131:40403), bf36ce71d6e542a4ac6f936f8dc2f9da (127.2.243.130:36443)
Jun 20 11:35:53 dist-test-slave-ctbw krb5kdc[16730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955351, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.131@KRBTEST.COM
Jun 20 11:35:53 dist-test-slave-ctbw krb5kdc[16730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955351, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.130@KRBTEST.COM
I20260620 11:35:53.086009 18299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:53.075304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:40403 (local address 127.2.243.129:35065)
0620 11:35:53.075385 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:53.075401 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:53.075475 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:53.076357 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:53.076360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:53.076779 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:53.076972 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:53.076978 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:53.078090 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:53.078091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:53.078904 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:53.078907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:53.079539 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:53.079540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:53.079651 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:53.080326 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:53.080330 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:53.081881 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:53.084233 (+ 2352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:53.084240 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:53.084242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:53.084545 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:53.084869 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:53.084870 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:53.084870 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:53.084997 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:53.085441 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:53.085447 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:53.085591 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 11:35:53.085699 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 11:35:53.085907 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":17}
I20260620 11:35:53.086043 18304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:53.075536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:41713 (local address 127.2.243.130:36443)
0620 11:35:53.075647 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:53.075657 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:53.075691 (+ 34us) server_negotiation.cc:407] Connection header received
0620 11:35:53.076139 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:53.076141 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:53.076179 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:53.076331 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:53.077093 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:53.077950 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:53.079044 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:53.079412 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:53.079558 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:53.082404 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:53.082407 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:53.082407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:53.082417 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:53.084404 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:53.084877 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:53.084880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:53.084880 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:53.085022 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:53.085214 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:53.085214 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:53.085215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:53.085453 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:53.085533 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:53.085786 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 11:35:53.085907 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":53,"threads_started":1}
I20260620 11:35:53.086251 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:53.075409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:36443 (local address 127.2.243.129:41713)
0620 11:35:53.075553 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:53.075564 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:35:53.075622 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:35:53.076357 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:35:53.076360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:35:53.076781 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:35:53.076972 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:53.076978 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:53.078066 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:53.078067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:53.078858 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:35:53.078904 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:35:53.079542 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:35:53.079543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:35:53.079651 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:53.080310 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:35:53.080318 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:35:53.082304 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:35:53.084524 (+ 2220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:53.084525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:53.084525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:53.084791 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:53.085079 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:35:53.085080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:35:53.085081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:35:53.085144 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:35:53.085538 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:35:53.085539 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:35:53.085591 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 11:35:53.085699 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 11:35:53.085989 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":208,"thread_start_us":77,"threads_started":1}
I20260620 11:35:53.086040 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:53.075431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:35065 (local address 127.2.243.131:40403)
0620 11:35:53.075595 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 11:35:53.075608 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 11:35:53.075638 (+ 30us) server_negotiation.cc:407] Connection header received
0620 11:35:53.076114 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:35:53.076116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:35:53.076170 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:35:53.076331 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:35:53.077116 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:53.077969 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:53.078993 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:35:53.079411 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:35:53.079552 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:35:53.081999 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:35:53.082004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:35:53.082005 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:35:53.082017 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:35:53.084131 (+ 2114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:53.084668 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:53.084671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:53.084672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:53.084789 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:35:53.085076 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:35:53.085077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:35:53.085077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:35:53.085369 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:35:53.085431 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 11:35:53.085765 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 11:35:53.085892 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
W20260620 11:35:53.087321 17916 leader_election.cc:341] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf36ce71d6e542a4ac6f936f8dc2f9da (127.2.243.130:36443): Not found: Tablet not found: 0a5c3b1120e74b20beb1395b44049278
W20260620 11:35:53.087422 17916 leader_election.cc:341] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58e345277f5444f7aa9d70d1411e76b9 (127.2.243.131:40403): Not found: Tablet not found: 0a5c3b1120e74b20beb1395b44049278
I20260620 11:35:53.087458 17916 leader_election.cc:302] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7371fe0ea9e04b2196329df942c10b7d; no voters: 58e345277f5444f7aa9d70d1411e76b9, bf36ce71d6e542a4ac6f936f8dc2f9da
I20260620 11:35:53.087587 18301 raft_consensus.cc:2726] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 11:35:53.091745 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:53.091521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.131:41999)
0620 11:35:53.091668 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:53.091712 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":104,"threads_started":1}
I20260620 11:35:53.893937 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:53.893661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.129:59351)
0620 11:35:53.893825 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:53.893880 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260620 11:35:53.968609 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:53.968355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.130:36419)
0620 11:35:53.968519 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:53.968563 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":104,"threads_started":1}
I20260620 11:35:54.092456 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:54.092193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.131:48585)
0620 11:35:54.092366 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:54.092411 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
I20260620 11:35:54.894706 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:54.894436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.129:40049)
0620 11:35:54.894604 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:54.894658 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":65,"threads_started":1}
I20260620 11:35:54.969336 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:54.969092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.130:55999)
0620 11:35:54.969258 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:54.969297 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":63,"threads_started":1}
I20260620 11:35:55.093232 18311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:55.092975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.131:55081)
0620 11:35:55.093143 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:55.093186 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":75,"threads_started":1}
I20260620 11:35:55.146929 18312 raft_consensus.cc:482] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:35:55.147033 18312 raft_consensus.cc:504] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } } peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } }
I20260620 11:35:55.147217 18312 leader_election.cc:288] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58e345277f5444f7aa9d70d1411e76b9 (127.2.243.131:40403), bf36ce71d6e542a4ac6f936f8dc2f9da (127.2.243.130:36443)
W20260620 11:35:55.147676 17916 leader_election.cc:341] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58e345277f5444f7aa9d70d1411e76b9 (127.2.243.131:40403): Not found: Tablet not found: 0a5c3b1120e74b20beb1395b44049278
W20260620 11:35:55.147774 17916 leader_election.cc:341] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf36ce71d6e542a4ac6f936f8dc2f9da (127.2.243.130:36443): Not found: Tablet not found: 0a5c3b1120e74b20beb1395b44049278
I20260620 11:35:55.147807 17916 leader_election.cc:302] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7371fe0ea9e04b2196329df942c10b7d; no voters: 58e345277f5444f7aa9d70d1411e76b9, bf36ce71d6e542a4ac6f936f8dc2f9da
I20260620 11:35:55.147902 18312 raft_consensus.cc:2726] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 11:35:55.895450 18313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:55.895227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.129:36019)
0620 11:35:55.895343 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:55.895393 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260620 11:35:55.970005 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:55.969800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.130:52733)
0620 11:35:55.969925 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:55.969960 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":57,"threads_started":1}
I20260620 11:35:56.094129 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:56.093874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.131:45695)
0620 11:35:56.094038 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:56.094084 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":115,"threads_started":1}
I20260620 11:35:56.881294 18316 raft_consensus.cc:482] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:35:56.881393 18316 raft_consensus.cc:504] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } } peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } }
I20260620 11:35:56.881597 18316 leader_election.cc:288] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58e345277f5444f7aa9d70d1411e76b9 (127.2.243.131:40403), bf36ce71d6e542a4ac6f936f8dc2f9da (127.2.243.130:36443)
W20260620 11:35:56.882063 17916 leader_election.cc:341] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf36ce71d6e542a4ac6f936f8dc2f9da (127.2.243.130:36443): Not found: Tablet not found: 0a5c3b1120e74b20beb1395b44049278
W20260620 11:35:56.882128 17916 leader_election.cc:341] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58e345277f5444f7aa9d70d1411e76b9 (127.2.243.131:40403): Not found: Tablet not found: 0a5c3b1120e74b20beb1395b44049278
I20260620 11:35:56.882144 17916 leader_election.cc:302] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7371fe0ea9e04b2196329df942c10b7d; no voters: 58e345277f5444f7aa9d70d1411e76b9, bf36ce71d6e542a4ac6f936f8dc2f9da
I20260620 11:35:56.882254 18316 raft_consensus.cc:2726] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 11:35:56.896289 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:56.895957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.129:45273)
0620 11:35:56.896164 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:56.896234 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":62,"threads_started":1}
I20260620 11:35:56.970635 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:56.970429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.130:36479)
0620 11:35:56.970558 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:56.970594 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
I20260620 11:35:57.094820 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:57.094596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.131:36627)
0620 11:35:57.094719 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:57.094762 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260620 11:35:57.896935 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:57.896733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.129:58315)
0620 11:35:57.896849 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:57.896890 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260620 11:35:57.971374 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:57.971132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.130:42815)
0620 11:35:57.971267 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:57.971310 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260620 11:35:58.095569 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:58.095345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.131:59829)
0620 11:35:58.095480 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:58.095523 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260620 11:35:58.897708 18323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:58.897481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.129:54061)
0620 11:35:58.897629 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:58.897664 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":90,"threads_started":1}
I20260620 11:35:58.972190 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:58.971940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.130:34197)
0620 11:35:58.972106 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:58.972145 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260620 11:35:59.096351 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:59.096128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.131:45359)
0620 11:35:59.096265 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:59.096305 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260620 11:35:59.192188 18326 raft_consensus.cc:482] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:35:59.192277 18326 raft_consensus.cc:504] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } } peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } }
I20260620 11:35:59.192428 18326 leader_election.cc:288] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58e345277f5444f7aa9d70d1411e76b9 (127.2.243.131:40403), bf36ce71d6e542a4ac6f936f8dc2f9da (127.2.243.130:36443)
W20260620 11:35:59.192919 17916 leader_election.cc:341] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf36ce71d6e542a4ac6f936f8dc2f9da (127.2.243.130:36443): Not found: Tablet not found: 0a5c3b1120e74b20beb1395b44049278
W20260620 11:35:59.192982 17916 leader_election.cc:341] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58e345277f5444f7aa9d70d1411e76b9 (127.2.243.131:40403): Not found: Tablet not found: 0a5c3b1120e74b20beb1395b44049278
I20260620 11:35:59.192997 17916 leader_election.cc:302] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7371fe0ea9e04b2196329df942c10b7d; no voters: 58e345277f5444f7aa9d70d1411e76b9, bf36ce71d6e542a4ac6f936f8dc2f9da
I20260620 11:35:59.193112 18326 raft_consensus.cc:2726] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 11:35:59.898347 18327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:59.898139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.129:60277)
0620 11:35:59.898262 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:59.898302 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260620 11:35:59.972849 18328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:35:59.972647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.130:34409)
0620 11:35:59.972765 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 11:35:59.972801 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260620 11:36:00.097044 18329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:00.096819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.131:52515)
0620 11:36:00.096963 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:00.097000 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":61,"threads_started":1}
I20260620 11:36:00.899081 18330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:00.898812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.129:55927)
0620 11:36:00.898982 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:00.899028 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":116,"threads_started":1}
I20260620 11:36:00.973515 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:00.973307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.130:37173)
0620 11:36:00.973427 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:00.973470 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260620 11:36:01.097815 18332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:01.097592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.131:43989)
0620 11:36:01.097720 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:01.097766 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260620 11:36:01.899839 18333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:01.899592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.129:41455)
0620 11:36:01.899730 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:01.899779 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":58,"threads_started":1}
I20260620 11:36:01.974218 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:01.973982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.130:33121)
0620 11:36:01.974131 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:01.974171 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
I20260620 11:36:02.098537 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:02.098336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.131:33287)
0620 11:36:02.098456 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:02.098492 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260620 11:36:02.900732 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:02.900490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.129:50873)
0620 11:36:02.900634 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:02.900686 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
I20260620 11:36:02.975138 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:02.974856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.130:55317)
0620 11:36:02.975040 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:02.975090 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":71,"threads_started":1}
I20260620 11:36:03.099391 18338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:03.099138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.131:60991)
0620 11:36:03.099297 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:03.099343 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":64,"threads_started":1}
I20260620 11:36:03.901721 18339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:03.901454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.129:58433)
0620 11:36:03.901619 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:03.901673 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":103,"threads_started":1}
I20260620 11:36:03.975945 18340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:03.975702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.130:43251)
0620 11:36:03.975855 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:03.975899 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":79,"threads_started":1}
I20260620 11:36:04.100102 18341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:04.099852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.131:60697)
0620 11:36:04.099997 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:04.100040 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":60,"threads_started":1}
I20260620 11:36:04.902465 18342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:04.902225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.129:48381)
0620 11:36:04.902378 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:04.902420 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260620 11:36:04.976648 18343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:04.976431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.130:40993)
0620 11:36:04.976565 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:04.976603 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260620 11:36:05.100919 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:05.100688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.131:57887)
0620 11:36:05.100837 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:05.100872 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260620 11:36:05.602600 18345 raft_consensus.cc:482] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:36:05.602730 18345 raft_consensus.cc:504] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58e345277f5444f7aa9d70d1411e76b9" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 40403 } } peers { permanent_uuid: "7371fe0ea9e04b2196329df942c10b7d" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 39513 } } peers { permanent_uuid: "bf36ce71d6e542a4ac6f936f8dc2f9da" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 36443 } }
I20260620 11:36:05.602963 18345 leader_election.cc:288] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 58e345277f5444f7aa9d70d1411e76b9 (127.2.243.131:40403), bf36ce71d6e542a4ac6f936f8dc2f9da (127.2.243.130:36443)
W20260620 11:36:05.603423 17916 leader_election.cc:341] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bf36ce71d6e542a4ac6f936f8dc2f9da (127.2.243.130:36443): Not found: Tablet not found: 0a5c3b1120e74b20beb1395b44049278
W20260620 11:36:05.603482 17916 leader_election.cc:341] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 58e345277f5444f7aa9d70d1411e76b9 (127.2.243.131:40403): Not found: Tablet not found: 0a5c3b1120e74b20beb1395b44049278
I20260620 11:36:05.603500 17916 leader_election.cc:302] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7371fe0ea9e04b2196329df942c10b7d; no voters: 58e345277f5444f7aa9d70d1411e76b9, bf36ce71d6e542a4ac6f936f8dc2f9da
I20260620 11:36:05.603572 18345 raft_consensus.cc:2726] T 0a5c3b1120e74b20beb1395b44049278 P 7371fe0ea9e04b2196329df942c10b7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 11:36:05.903247 18346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:05.903006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.129:54307)
0620 11:36:05.903155 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:05.903199 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":72,"threads_started":1}
I20260620 11:36:05.977382 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:05.977150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.130:46413)
0620 11:36:05.977285 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:05.977332 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":59,"threads_started":1}
I20260620 11:36:06.101635 18348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:06.101391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.131:36233)
0620 11:36:06.101539 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:06.101585 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
I20260620 11:36:06.904018 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:06.903776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.129:54889)
0620 11:36:06.903904 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:06.903961 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260620 11:36:06.978199 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:06.977976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.130:56171)
0620 11:36:06.978115 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:06.978153 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260620 11:36:07.100021 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 17870
I20260620 11:36:07.102437 18351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:07.102200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:43247 (local address 127.2.243.131:60119)
0620 11:36:07.102359 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:07.102396 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:43247: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":66,"threads_started":1}
I20260620 11:36:07.105000 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 18032
I20260620 11:36:07.109237 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 18166
2026-06-20T11:36:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18183 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:36:07 dist-test-slave-ctbw krb5kdc[18354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:36:07 dist-test-slave-ctbw krb5kdc[18354](info): set up 2 sockets
Jun 20 11:36:07 dist-test-slave-ctbw krb5kdc[18354](info): commencing operation
krb5kdc: starting...
W20260620 11:36:08.147184 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:36:08 dist-test-slave-ctbw krb5kdc[18354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:36:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:36:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:36:08.262161 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:40113
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:40113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:36:08.344555 18367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:36:08.344799 18367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:36:08.344844 18367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:36:08.345999 18367 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 11:36:08.346064 18367 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 11:36:08.346086 18367 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 11:36:08.346105 18367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:36:08.346138 18367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:36:08.346164 18367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:36:08.346194 18367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:36:08.346225 18367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:36:08.347738 18367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:40113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:40113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:36:08.348018 18367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:36:08.348244 18367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:36:08.353581 18372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:36:08.353570 18375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:36:08.353638 18367 server_base.cc:1034] running on GCE node
W20260620 11:36:08.353574 18373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:36:08.354070 18367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:36:08.354327 18367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:36:08.355468 18367 hybrid_clock.cc:648] HybridClock initialized: now 1781955368355449 us; error 42 us; skew 500 ppm
Jun 20 11:36:08 dist-test-slave-ctbw krb5kdc[18354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955368, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:36:08.357954 18367 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:36:08.358330 18367 webserver.cc:460] Webserver started at http://127.2.243.190:36629/ using document root <none> and password file <none>
I20260620 11:36:08.358521 18367 fs_manager.cc:359] Metadata directory not provided
I20260620 11:36:08.358562 18367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:36:08.358649 18367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:36:08.359732 18367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "85eba415a72a4fc7ad11ac4d98270ae0"
format_stamp: "Formatted at 2026-06-20 11:36:08 on dist-test-slave-ctbw"
server_key: "a713d011fee0dbc9199d4e2760253adc"
server_key_iv: "53060bef3130c5080ed92b884ebbf9ab"
server_key_version: "encryptionkey@0"
I20260620 11:36:08.360033 18367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "85eba415a72a4fc7ad11ac4d98270ae0"
format_stamp: "Formatted at 2026-06-20 11:36:08 on dist-test-slave-ctbw"
server_key: "a713d011fee0dbc9199d4e2760253adc"
server_key_iv: "53060bef3130c5080ed92b884ebbf9ab"
server_key_version: "encryptionkey@0"
I20260620 11:36:08.361301 18367 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 11:36:08.361987 18382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:08.362141 18367 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:36:08.362224 18367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "85eba415a72a4fc7ad11ac4d98270ae0"
format_stamp: "Formatted at 2026-06-20 11:36:08 on dist-test-slave-ctbw"
server_key: "a713d011fee0dbc9199d4e2760253adc"
server_key_iv: "53060bef3130c5080ed92b884ebbf9ab"
server_key_version: "encryptionkey@0"
I20260620 11:36:08.362280 18367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:36:08.370002 18367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:36:08.372268 18367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:36:08.372377 18367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:36:08.376134 18367 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:40113
I20260620 11:36:08.376199 18434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:40113 every 8 connection(s)
I20260620 11:36:08.376546 18367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:36:08.376693 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 18367
I20260620 11:36:08.376814 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:36:08.377054 3022 external_mini_cluster.cc:1428] Setting key 8d39fa3bd4caf1e333b7640d4a0f10f6
I20260620 11:36:08.377496 18435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:36:08.380290 18435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0: Bootstrap starting.
I20260620 11:36:08.380877 18435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0: Neither blocks nor log segments found. Creating new log.
I20260620 11:36:08.381179 18435 log.cc:824] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0: Log is configured to *not* fsync() on all Append() calls
I20260620 11:36:08.381942 18435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0: No bootstrap required, opened a new log
Jun 20 11:36:08 dist-test-slave-ctbw krb5kdc[18354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:36:08.384243 18435 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40113 } }
I20260620 11:36:08.384405 18435 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:36:08.384454 18435 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85eba415a72a4fc7ad11ac4d98270ae0, State: Initialized, Role: FOLLOWER
I20260620 11:36:08.384586 18435 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40113 } }
I20260620 11:36:08.384680 18435 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:36:08.384724 18435 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:36:08.384771 18435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:36:08.385277 18435 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40113 } }
I20260620 11:36:08.385401 18435 leader_election.cc:302] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85eba415a72a4fc7ad11ac4d98270ae0; no voters:
I20260620 11:36:08.385592 18435 leader_election.cc:288] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:36:08.385648 18440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:36:08.385839 18440 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 1 LEADER]: Becoming Leader. State: Replica: 85eba415a72a4fc7ad11ac4d98270ae0, State: Running, Role: LEADER
I20260620 11:36:08.385849 18435 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:36:08.385954 18440 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40113 } }
I20260620 11:36:08.386267 18440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40113 } } }
I20260620 11:36:08.386370 18440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [sys.catalog]: This master's current role is: LEADER
I20260620 11:36:08.386505 18440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85eba415a72a4fc7ad11ac4d98270ae0. Latest consensus state: current_term: 1 leader_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40113 } } }
I20260620 11:36:08.386579 18440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [sys.catalog]: This master's current role is: LEADER
I20260620 11:36:08.386749 18446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:36:08.387275 18446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:36:08.388391 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:08.377599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42126 (local address 127.2.243.190:40113)
0620 11:36:08.377785 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:08.377793 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:08.377817 (+ 24us) server_negotiation.cc:407] Connection header received
0620 11:36:08.378261 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:08.378267 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:08.378310 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:08.378458 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:36:08.379152 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.380019 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.380844 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.381344 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.381470 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:08.383946 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:36:08.383952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:36:08.383952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:36:08.383966 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:36:08.386123 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:08.386560 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:08.386563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:08.386563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:08.386680 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:08.386942 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:08.386943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:08.386944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:08.387255 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:36:08.387444 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:08.388049 (+ 605us) server_negotiation.cc:299] Negotiation successful
0620 11:36:08.388201 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
I20260620 11:36:08.389479 18446 catalog_manager.cc:1347] Generated new cluster ID: 1ed260a67ba7455e87e8c8e63f440937
I20260620 11:36:08.389528 18446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:36:08.419307 18446 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:36:08.420089 18446 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:36:08.423785 18446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0: Generated new TSK 0
I20260620 11:36:08.423977 18446 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
I20260620 11:36:08.453136 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:40113
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:36:08.534049 18461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:36:08.534255 18461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:36:08.534278 18461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:36:08.535481 18461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:36:08.535526 18461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:36:08.535552 18461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:36:08.536984 18461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:40113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:36:08.537226 18461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:36:08.537459 18461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:36:08.542479 18469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:36:08.542486 18466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:36:08.542687 18461 server_base.cc:1034] running on GCE node
W20260620 11:36:08.542629 18467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:36:08.543009 18461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:36:08.543262 18461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:36:08.544432 18461 hybrid_clock.cc:648] HybridClock initialized: now 1781955368544419 us; error 49 us; skew 500 ppm
Jun 20 11:36:08 dist-test-slave-ctbw krb5kdc[18354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955368, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:36:08.546833 18461 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:36:08.547278 18461 webserver.cc:460] Webserver started at http://127.2.243.129:34769/ using document root <none> and password file <none>
I20260620 11:36:08.547509 18461 fs_manager.cc:359] Metadata directory not provided
I20260620 11:36:08.547578 18461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:36:08.547711 18461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:36:08.548743 18461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b"
format_stamp: "Formatted at 2026-06-20 11:36:08 on dist-test-slave-ctbw"
server_key: "d6b148c71b660d8814cf3879ce5ad72e"
server_key_iv: "903d8eec430bd277016f1a3a5184537d"
server_key_version: "encryptionkey@0"
I20260620 11:36:08.549117 18461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b"
format_stamp: "Formatted at 2026-06-20 11:36:08 on dist-test-slave-ctbw"
server_key: "d6b148c71b660d8814cf3879ce5ad72e"
server_key_iv: "903d8eec430bd277016f1a3a5184537d"
server_key_version: "encryptionkey@0"
I20260620 11:36:08.550421 18461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:36:08.551210 18476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:08.551385 18461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:36:08.551484 18461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b"
format_stamp: "Formatted at 2026-06-20 11:36:08 on dist-test-slave-ctbw"
server_key: "d6b148c71b660d8814cf3879ce5ad72e"
server_key_iv: "903d8eec430bd277016f1a3a5184537d"
server_key_version: "encryptionkey@0"
I20260620 11:36:08.551568 18461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:36:08.570295 18461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:36:08.572430 18461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:36:08.572593 18461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:36:08.572822 18461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:36:08.573128 18461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:36:08.573184 18461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:08.573223 18461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:36:08.573251 18461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:08.579708 18461 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:32799
I20260620 11:36:08.579813 18589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:32799 every 8 connection(s)
I20260620 11:36:08.580125 18461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
Jun 20 11:36:08 dist-test-slave-ctbw krb5kdc[18354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955368, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:36:08.588483 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 18461
I20260620 11:36:08.588603 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:36:08.588850 3022 external_mini_cluster.cc:1428] Setting key fc9b62ed314c27a23ee51253e470fd04
I20260620 11:36:08.589648 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:08.580895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:51585 (local address 127.2.243.190:40113)
0620 11:36:08.580946 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:08.580948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:08.581131 (+ 183us) server_negotiation.cc:407] Connection header received
0620 11:36:08.582108 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:08.582110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:08.582122 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:08.582200 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:36:08.582822 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.583390 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.584343 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.584556 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.584625 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:08.586751 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:36:08.586753 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:36:08.586753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:36:08.586761 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:36:08.588159 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:08.588545 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:08.588547 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:08.588547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:08.588619 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:08.588907 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:08.588908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:08.588908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:08.589058 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:36:08.589130 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:08.589445 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 11:36:08.589550 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:36:08.589859 18592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:08.580844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.129:51585)
0620 11:36:08.580997 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:08.581039 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:36:08.581989 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:36:08.582252 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:36:08.582255 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:36:08.582513 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:36:08.582703 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:08.582707 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.583472 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.583473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:08.584260 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:08.584262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.584642 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.584642 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:08.584739 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:08.585176 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:36:08.585184 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:36:08.586659 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:36:08.588231 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:36:08.588238 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:36:08.588239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:36:08.588466 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:36:08.588711 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:36:08.588712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:36:08.588712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:36:08.588831 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:36:08.589133 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:36:08.589140 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:36:08.589306 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 11:36:08.589421 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 11:36:08.589591 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
I20260620 11:36:08.590343 18590 heartbeater.cc:347] Connected to a master server at 127.2.243.190:40113
I20260620 11:36:08.590442 18590 heartbeater.cc:464] Registering TS with master...
I20260620 11:36:08.590624 18590 heartbeater.cc:511] Master 127.2.243.190:40113 requested a full tablet report, sending...
I20260620 11:36:08.591123 18399 ts_manager.cc:194] Registered new tserver with Master: 59c49a3f678c4ebdb8aa4063e5f31a2b (127.2.243.129:32799)
I20260620 11:36:08.591714 18399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:51585
WARNING: no policy specified for kudu/127.2.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
I20260620 11:36:08.612876 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:40113
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:36:08.694810 18595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:36:08.695026 18595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:36:08.695050 18595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:36:08.696213 18595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:36:08.696255 18595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:36:08.696280 18595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:36:08.697803 18595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:40113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:36:08.698040 18595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:36:08.698266 18595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:36:08.703423 18600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:36:08.703413 18601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:36:08.703490 18595 server_base.cc:1034] running on GCE node
W20260620 11:36:08.703418 18603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:36:08.703994 18595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:36:08.704296 18595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:36:08.705470 18595 hybrid_clock.cc:648] HybridClock initialized: now 1781955368705457 us; error 46 us; skew 500 ppm
Jun 20 11:36:08 dist-test-slave-ctbw krb5kdc[18354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955368, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:36:08.708107 18595 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:36:08.708563 18595 webserver.cc:460] Webserver started at http://127.2.243.130:36341/ using document root <none> and password file <none>
I20260620 11:36:08.708750 18595 fs_manager.cc:359] Metadata directory not provided
I20260620 11:36:08.708791 18595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:36:08.708877 18595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:36:08.709860 18595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "ba680ca73e064a0cbde4f12e7c2c4080"
format_stamp: "Formatted at 2026-06-20 11:36:08 on dist-test-slave-ctbw"
server_key: "e086db3aed0c76b25b9b7115a468500d"
server_key_iv: "cf2d1f3b7efb8c6d5bb68ba680e5b5dc"
server_key_version: "encryptionkey@0"
I20260620 11:36:08.710183 18595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "ba680ca73e064a0cbde4f12e7c2c4080"
format_stamp: "Formatted at 2026-06-20 11:36:08 on dist-test-slave-ctbw"
server_key: "e086db3aed0c76b25b9b7115a468500d"
server_key_iv: "cf2d1f3b7efb8c6d5bb68ba680e5b5dc"
server_key_version: "encryptionkey@0"
I20260620 11:36:08.711530 18595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:36:08.712201 18610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:08.712395 18595 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:36:08.712488 18595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "ba680ca73e064a0cbde4f12e7c2c4080"
format_stamp: "Formatted at 2026-06-20 11:36:08 on dist-test-slave-ctbw"
server_key: "e086db3aed0c76b25b9b7115a468500d"
server_key_iv: "cf2d1f3b7efb8c6d5bb68ba680e5b5dc"
server_key_version: "encryptionkey@0"
I20260620 11:36:08.712561 18595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:36:08.731060 18595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:36:08.733119 18595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:36:08.733294 18595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:36:08.733544 18595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:36:08.733866 18595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:36:08.733899 18595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:08.733922 18595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:36:08.733973 18595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:08.740129 18595 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:39745
I20260620 11:36:08.740188 18723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:39745 every 8 connection(s)
I20260620 11:36:08.740517 18595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
Jun 20 11:36:08 dist-test-slave-ctbw krb5kdc[18354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955368, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:36:08.747877 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 18595
I20260620 11:36:08.748013 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:36:08.748207 3022 external_mini_cluster.cc:1428] Setting key caacf110c7265c9871b15b3f8e427a27
I20260620 11:36:08.750043 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:08.741251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:42291 (local address 127.2.243.190:40113)
0620 11:36:08.741322 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:08.741325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:08.741503 (+ 178us) server_negotiation.cc:407] Connection header received
0620 11:36:08.742038 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:08.742040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:08.742051 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:08.742103 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:36:08.743116 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.743620 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.744642 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.744841 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.744927 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:08.747123 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:36:08.747125 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:36:08.747125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:36:08.747132 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:36:08.748659 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:08.749130 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:08.749131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:08.749131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:08.749181 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:08.749449 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:08.749449 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:08.749450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:08.749541 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:36:08.749609 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:08.749920 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 11:36:08.749980 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 11:36:08.750336 18726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:08.741234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.130:42291)
0620 11:36:08.741385 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:08.741421 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 11:36:08.741926 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:36:08.742192 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:36:08.742196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:36:08.742762 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:36:08.743021 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:08.743027 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.743740 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.743741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:08.744551 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:08.744553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.744936 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.744936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:08.745056 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:08.745536 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:36:08.745544 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:36:08.747036 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:36:08.748729 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:36:08.748735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:36:08.748737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:36:08.749049 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:36:08.749240 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:36:08.749241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:36:08.749241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:36:08.749384 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:36:08.749600 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:36:08.749607 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:36:08.749783 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 11:36:08.749896 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 11:36:08.750074 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
I20260620 11:36:08.751076 18724 heartbeater.cc:347] Connected to a master server at 127.2.243.190:40113
I20260620 11:36:08.751158 18724 heartbeater.cc:464] Registering TS with master...
I20260620 11:36:08.751368 18724 heartbeater.cc:511] Master 127.2.243.190:40113 requested a full tablet report, sending...
I20260620 11:36:08.751808 18399 ts_manager.cc:194] Registered new tserver with Master: ba680ca73e064a0cbde4f12e7c2c4080 (127.2.243.130:39745)
I20260620 11:36:08.752184 18399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:42291
WARNING: no policy specified for kudu/127.2.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
I20260620 11:36:08.773149 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:40113
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:36:08.853839 18729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:36:08.854082 18729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:36:08.854125 18729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:36:08.855441 18729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:36:08.855506 18729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:36:08.855551 18729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:36:08.857014 18729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:40113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:36:08.857291 18729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:36:08.857532 18729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:36:08.862515 18737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:36:08.862591 18734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:36:08.862514 18735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:36:08.862754 18729 server_base.cc:1034] running on GCE node
I20260620 11:36:08.863037 18729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:36:08.863286 18729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:36:08.864481 18729 hybrid_clock.cc:648] HybridClock initialized: now 1781955368864476 us; error 43 us; skew 500 ppm
Jun 20 11:36:08 dist-test-slave-ctbw krb5kdc[18354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955368, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:36:08.867205 18729 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:36:08.867640 18729 webserver.cc:460] Webserver started at http://127.2.243.131:35113/ using document root <none> and password file <none>
I20260620 11:36:08.867888 18729 fs_manager.cc:359] Metadata directory not provided
I20260620 11:36:08.867956 18729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:36:08.868098 18729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:36:08.869185 18729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "d7b3217648914448b7eaa923a4e7d3d3"
format_stamp: "Formatted at 2026-06-20 11:36:08 on dist-test-slave-ctbw"
server_key: "2731c599644d514da4476b31863bb525"
server_key_iv: "4bc261953bb41bcf6ded55b304ee65d4"
server_key_version: "encryptionkey@0"
I20260620 11:36:08.869542 18729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "d7b3217648914448b7eaa923a4e7d3d3"
format_stamp: "Formatted at 2026-06-20 11:36:08 on dist-test-slave-ctbw"
server_key: "2731c599644d514da4476b31863bb525"
server_key_iv: "4bc261953bb41bcf6ded55b304ee65d4"
server_key_version: "encryptionkey@0"
I20260620 11:36:08.871129 18729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 11:36:08.872215 18744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:08.872440 18729 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:36:08.872530 18729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "d7b3217648914448b7eaa923a4e7d3d3"
format_stamp: "Formatted at 2026-06-20 11:36:08 on dist-test-slave-ctbw"
server_key: "2731c599644d514da4476b31863bb525"
server_key_iv: "4bc261953bb41bcf6ded55b304ee65d4"
server_key_version: "encryptionkey@0"
I20260620 11:36:08.872599 18729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:36:08.890156 18729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:36:08.892652 18729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:36:08.892818 18729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:36:08.893035 18729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:36:08.893345 18729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:36:08.893399 18729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:08.893441 18729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:36:08.893468 18729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:08.899307 18729 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:45031
I20260620 11:36:08.899390 18857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:45031 every 8 connection(s)
I20260620 11:36:08.899703 18729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
Jun 20 11:36:08 dist-test-slave-ctbw krb5kdc[18354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955368, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:36:08.908155 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 18729
I20260620 11:36:08.908268 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:36:08.908476 3022 external_mini_cluster.cc:1428] Setting key 0d1befb34e677b678e6d411bac119f0f
I20260620 11:36:08.909243 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:08.900417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:58915 (local address 127.2.243.190:40113)
0620 11:36:08.900462 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:08.900464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:08.900651 (+ 187us) server_negotiation.cc:407] Connection header received
0620 11:36:08.901215 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:08.901217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:08.901231 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:08.901288 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:36:08.902244 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.902753 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.903796 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.904015 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.904112 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:08.906288 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:36:08.906291 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:36:08.906291 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:36:08.906299 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:36:08.907629 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:08.908080 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:08.908082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:08.908083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:08.908145 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:08.908417 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:08.908418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:08.908418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:08.908563 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:36:08.908631 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:08.909022 (+ 391us) server_negotiation.cc:299] Negotiation successful
0620 11:36:08.909139 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:36:08.909408 18860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:08.900383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.131:58915)
0620 11:36:08.900519 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:08.900564 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 11:36:08.901090 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:36:08.901358 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:36:08.901362 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:36:08.901937 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:36:08.902126 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:08.902132 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.902850 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.902851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:08.903699 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:08.903702 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.904118 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.904119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:08.904296 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:08.904815 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:36:08.904824 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:36:08.906187 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:36:08.907749 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:36:08.907757 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:36:08.907759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:36:08.907989 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:36:08.908232 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:36:08.908233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:36:08.908233 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:36:08.908364 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:36:08.908621 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:36:08.908627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:36:08.908841 (+ 214us) client_negotiation.cc:769] Sending connection context
0620 11:36:08.908975 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 11:36:08.909154 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":65,"threads_started":1}
I20260620 11:36:08.909859 18858 heartbeater.cc:347] Connected to a master server at 127.2.243.190:40113
I20260620 11:36:08.909965 18858 heartbeater.cc:464] Registering TS with master...
I20260620 11:36:08.910145 18858 heartbeater.cc:511] Master 127.2.243.190:40113 requested a full tablet report, sending...
I20260620 11:36:08.910516 18399 ts_manager.cc:194] Registered new tserver with Master: d7b3217648914448b7eaa923a4e7d3d3 (127.2.243.131:45031)
I20260620 11:36:08.910974 18399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:58915
I20260620 11:36:08.920539 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:36:08.928263 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:08.921994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42130 (local address 127.2.243.190:40113)
0620 11:36:08.922055 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:08.922057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:08.922156 (+ 99us) server_negotiation.cc:407] Connection header received
0620 11:36:08.922208 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:08.922209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:08.922219 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:08.922284 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:36:08.922800 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.923350 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.924159 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.924367 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.924445 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:08.925744 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:36:08.925745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:36:08.925745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:36:08.925751 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:36:08.927296 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:08.927598 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:08.927599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:08.927599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:08.927677 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:08.927832 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:08.927832 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:08.927833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:08.927966 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:36:08.928033 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:08.928140 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 11:36:08.928198 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 11:36:08.930275 18399 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:36:08.932035 18399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:36:08.938628 18870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:08.934324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:32799 (local address 127.0.0.1:45564)
0620 11:36:08.934854 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:08.934900 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 11:36:08.934938 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:36:08.935263 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:36:08.935265 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:36:08.935267 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:36:08.935298 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:08.935299 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.936482 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.936483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:08.937435 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:08.937437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.938306 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.938307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:08.938384 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:08.938402 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 11:36:08.938475 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 11:36:08.938533 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":2,"thread_start_us":96,"threads_started":1}
I20260620 11:36:08.938654 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:08.934538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45564 (local address 127.2.243.129:32799)
0620 11:36:08.935021 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:08.935026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:08.935035 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:36:08.935072 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:08.935074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:08.935089 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:08.935163 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:36:08.935383 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.936383 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.937521 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.938208 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.938283 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:08.938341 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:08.938503 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 11:36:08.938554 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":75,"threads_started":1}
I20260620 11:36:08.938815 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:08.934315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:39745 (local address 127.0.0.1:53738)
0620 11:36:08.934731 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:08.934741 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:36:08.934797 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:36:08.935554 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:36:08.935555 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:36:08.935556 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:36:08.935598 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:08.935600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.936534 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.936535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:08.937604 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:08.937607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.938400 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.938400 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:08.938459 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:08.938463 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:36:08.938508 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 11:36:08.938543 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":2,"thread_start_us":81,"threads_started":1}
I20260620 11:36:08.939036 18869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:08.934315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:45031 (local address 127.0.0.1:51996)
0620 11:36:08.934479 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:08.934518 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 11:36:08.934585 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:36:08.934852 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:36:08.934855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:36:08.934858 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:36:08.934954 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:08.934956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.936270 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.936271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:08.937692 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:08.937694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.938624 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.938625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:08.938694 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:08.938698 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:36:08.938911 (+ 213us) client_negotiation.cc:240] Negotiation successful
0620 11:36:08.938959 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":58,"threads_started":1}
I20260620 11:36:08.939335 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:08.934490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.2.243.131:45031)
0620 11:36:08.934681 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:08.934686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:08.934696 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:36:08.934741 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:08.934743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:08.934755 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:08.934847 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:36:08.935027 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.936156 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.937770 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.938471 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.938539 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:08.938588 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:08.939190 (+ 602us) server_negotiation.cc:299] Negotiation successful
0620 11:36:08.939237 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":82,"threads_started":1}
I20260620 11:36:08.939334 18874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:08.934563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53738 (local address 127.2.243.130:39745)
0620 11:36:08.935417 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:08.935421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:08.935428 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:36:08.935456 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:08.935458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:08.935469 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:08.935556 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:36:08.935655 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.936423 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.937700 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.938220 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.938309 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:08.938368 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:08.938583 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 11:36:08.938624 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":821,"thread_start_us":78,"threads_started":1}
I20260620 11:36:08.939546 18524 tablet_service.cc:1467] Processing CreateTablet for tablet a47127de9b3a490da12b201b2344cf90 (DEFAULT_TABLE table=test-table [id=0b12d6eb51ac4d9da22e82a54c06aa71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:36:08.939950 18524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a47127de9b3a490da12b201b2344cf90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:36:08.939971 18792 tablet_service.cc:1467] Processing CreateTablet for tablet a47127de9b3a490da12b201b2344cf90 (DEFAULT_TABLE table=test-table [id=0b12d6eb51ac4d9da22e82a54c06aa71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:36:08.940135 18658 tablet_service.cc:1467] Processing CreateTablet for tablet a47127de9b3a490da12b201b2344cf90 (DEFAULT_TABLE table=test-table [id=0b12d6eb51ac4d9da22e82a54c06aa71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:36:08.940321 18792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a47127de9b3a490da12b201b2344cf90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:36:08.940466 18658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a47127de9b3a490da12b201b2344cf90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:36:08.943416 18876 tablet_bootstrap.cc:492] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Bootstrap starting.
I20260620 11:36:08.943507 18875 tablet_bootstrap.cc:492] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3: Bootstrap starting.
I20260620 11:36:08.943994 18876 tablet_bootstrap.cc:654] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Neither blocks nor log segments found. Creating new log.
I20260620 11:36:08.944154 18875 tablet_bootstrap.cc:654] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3: Neither blocks nor log segments found. Creating new log.
I20260620 11:36:08.944545 18876 log.cc:824] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Log is configured to *not* fsync() on all Append() calls
I20260620 11:36:08.944684 18875 log.cc:824] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3: Log is configured to *not* fsync() on all Append() calls
I20260620 11:36:08.945037 18877 tablet_bootstrap.cc:492] T a47127de9b3a490da12b201b2344cf90 P ba680ca73e064a0cbde4f12e7c2c4080: Bootstrap starting.
I20260620 11:36:08.945401 18875 tablet_bootstrap.cc:492] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3: No bootstrap required, opened a new log
I20260620 11:36:08.945508 18875 ts_tablet_manager.cc:1397] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:36:08.945642 18877 tablet_bootstrap.cc:654] T a47127de9b3a490da12b201b2344cf90 P ba680ca73e064a0cbde4f12e7c2c4080: Neither blocks nor log segments found. Creating new log.
I20260620 11:36:08.946110 18877 log.cc:824] T a47127de9b3a490da12b201b2344cf90 P ba680ca73e064a0cbde4f12e7c2c4080: Log is configured to *not* fsync() on all Append() calls
I20260620 11:36:08.946751 18877 tablet_bootstrap.cc:492] T a47127de9b3a490da12b201b2344cf90 P ba680ca73e064a0cbde4f12e7c2c4080: No bootstrap required, opened a new log
I20260620 11:36:08.946803 18877 ts_tablet_manager.cc:1397] T a47127de9b3a490da12b201b2344cf90 P ba680ca73e064a0cbde4f12e7c2c4080: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:36:08.947098 18875 raft_consensus.cc:348] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:08.947271 18875 raft_consensus.cc:374] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:36:08.947321 18875 raft_consensus.cc:729] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b3217648914448b7eaa923a4e7d3d3, State: Initialized, Role: FOLLOWER
I20260620 11:36:08.947440 18875 consensus_queue.cc:260] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:08.947705 18875 ts_tablet_manager.cc:1428] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 11:36:08.947748 18858 heartbeater.cc:503] Master 127.2.243.190:40113 was elected leader, sending a full tablet report...
I20260620 11:36:08.948158 18876 tablet_bootstrap.cc:492] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b: No bootstrap required, opened a new log
I20260620 11:36:08.948235 18876 ts_tablet_manager.cc:1397] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 11:36:08.949177 18877 raft_consensus.cc:348] T a47127de9b3a490da12b201b2344cf90 P ba680ca73e064a0cbde4f12e7c2c4080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:08.949327 18877 raft_consensus.cc:374] T a47127de9b3a490da12b201b2344cf90 P ba680ca73e064a0cbde4f12e7c2c4080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:36:08.949373 18877 raft_consensus.cc:729] T a47127de9b3a490da12b201b2344cf90 P ba680ca73e064a0cbde4f12e7c2c4080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba680ca73e064a0cbde4f12e7c2c4080, State: Initialized, Role: FOLLOWER
I20260620 11:36:08.949481 18877 consensus_queue.cc:260] T a47127de9b3a490da12b201b2344cf90 P ba680ca73e064a0cbde4f12e7c2c4080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:08.949766 18877 ts_tablet_manager.cc:1428] T a47127de9b3a490da12b201b2344cf90 P ba680ca73e064a0cbde4f12e7c2c4080: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 11:36:08.949796 18724 heartbeater.cc:503] Master 127.2.243.190:40113 was elected leader, sending a full tablet report...
I20260620 11:36:08.950179 18876 raft_consensus.cc:348] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:08.950357 18876 raft_consensus.cc:374] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:36:08.950424 18876 raft_consensus.cc:729] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59c49a3f678c4ebdb8aa4063e5f31a2b, State: Initialized, Role: FOLLOWER
I20260620 11:36:08.950536 18876 consensus_queue.cc:260] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:08.950825 18876 ts_tablet_manager.cc:1428] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:36:08.950845 18590 heartbeater.cc:503] Master 127.2.243.190:40113 was elected leader, sending a full tablet report...
W20260620 11:36:08.991086 18725 tablet.cc:2369] T a47127de9b3a490da12b201b2344cf90 P ba680ca73e064a0cbde4f12e7c2c4080: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:36:08.995602 18881 raft_consensus.cc:482] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:36:08.995780 18881 raft_consensus.cc:504] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:08.996204 18881 leader_election.cc:288] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba680ca73e064a0cbde4f12e7c2c4080 (127.2.243.130:39745), 59c49a3f678c4ebdb8aa4063e5f31a2b (127.2.243.129:32799)
I20260620 11:36:09.000273 18874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:08.996569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:34879 (local address 127.2.243.130:39745)
0620 11:36:08.996619 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:08.996621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:08.996725 (+ 104us) server_negotiation.cc:407] Connection header received
0620 11:36:08.996799 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:08.996800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:08.996809 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:08.996864 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:36:08.997023 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.997744 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.999281 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.999865 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.999945 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:08.999954 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:09.000157 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 11:36:09.000205 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 11:36:09.000273 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:08.996591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:35901 (local address 127.2.243.129:32799)
0620 11:36:08.996633 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:08.996635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:08.996645 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:36:08.996672 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:08.996673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:08.996682 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:08.996764 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:36:08.996977 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.997750 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.999282 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.999865 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.999945 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:08.999954 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:09.000157 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 11:36:09.000205 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:36:09.000248 18860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:08.996418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:32799 (local address 127.2.243.131:35901)
0620 11:36:08.996486 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:08.996498 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:36:08.996551 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:36:08.996788 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:36:08.996789 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:36:08.996791 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:36:08.996892 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:08.996894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.997838 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.997839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:08.999166 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:08.999169 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.999957 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.999958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:09.000043 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:09.000045 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:36:09.000104 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 11:36:09.000151 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":107}
I20260620 11:36:09.000258 18884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:08.996494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:39745 (local address 127.2.243.131:34879)
0620 11:36:08.996654 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:08.996663 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:36:08.996720 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:36:08.996887 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:36:08.996888 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:36:08.996889 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:36:08.996948 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:08.996949 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.997838 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.997839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:08.999177 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:08.999178 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:08.999958 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:08.999958 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:09.000040 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:09.000044 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:36:09.000104 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 11:36:09.000153 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":65,"thread_start_us":110,"threads_started":1}
I20260620 11:36:09.000663 18544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47127de9b3a490da12b201b2344cf90" candidate_uuid: "d7b3217648914448b7eaa923a4e7d3d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" is_pre_election: true
I20260620 11:36:09.000851 18544 raft_consensus.cc:2454] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7b3217648914448b7eaa923a4e7d3d3 in term 0.
I20260620 11:36:09.000945 18678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47127de9b3a490da12b201b2344cf90" candidate_uuid: "d7b3217648914448b7eaa923a4e7d3d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" is_pre_election: true
I20260620 11:36:09.001077 18678 raft_consensus.cc:2454] T a47127de9b3a490da12b201b2344cf90 P ba680ca73e064a0cbde4f12e7c2c4080 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7b3217648914448b7eaa923a4e7d3d3 in term 0.
I20260620 11:36:09.001333 18747 leader_election.cc:302] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59c49a3f678c4ebdb8aa4063e5f31a2b, d7b3217648914448b7eaa923a4e7d3d3; no voters:
I20260620 11:36:09.001523 18881 raft_consensus.cc:2781] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:36:09.001605 18881 raft_consensus.cc:482] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:36:09.001629 18881 raft_consensus.cc:3037] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:36:09.002308 18881 raft_consensus.cc:504] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:09.002452 18881 leader_election.cc:288] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3 [CANDIDATE]: Term 1 election: Requested vote from peers ba680ca73e064a0cbde4f12e7c2c4080 (127.2.243.130:39745), 59c49a3f678c4ebdb8aa4063e5f31a2b (127.2.243.129:32799)
I20260620 11:36:09.002599 18678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47127de9b3a490da12b201b2344cf90" candidate_uuid: "d7b3217648914448b7eaa923a4e7d3d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba680ca73e064a0cbde4f12e7c2c4080"
I20260620 11:36:09.002635 18544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a47127de9b3a490da12b201b2344cf90" candidate_uuid: "d7b3217648914448b7eaa923a4e7d3d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b"
I20260620 11:36:09.002682 18678 raft_consensus.cc:3037] T a47127de9b3a490da12b201b2344cf90 P ba680ca73e064a0cbde4f12e7c2c4080 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:36:09.002700 18544 raft_consensus.cc:3037] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:36:09.003611 18678 raft_consensus.cc:2454] T a47127de9b3a490da12b201b2344cf90 P ba680ca73e064a0cbde4f12e7c2c4080 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b3217648914448b7eaa923a4e7d3d3 in term 1.
I20260620 11:36:09.003621 18544 raft_consensus.cc:2454] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7b3217648914448b7eaa923a4e7d3d3 in term 1.
I20260620 11:36:09.003793 18745 leader_election.cc:302] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba680ca73e064a0cbde4f12e7c2c4080, d7b3217648914448b7eaa923a4e7d3d3; no voters:
I20260620 11:36:09.003876 18881 raft_consensus.cc:2781] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:36:09.004024 18881 raft_consensus.cc:686] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3 [term 1 LEADER]: Becoming Leader. State: Replica: d7b3217648914448b7eaa923a4e7d3d3, State: Running, Role: LEADER
I20260620 11:36:09.004127 18881 consensus_queue.cc:237] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:09.005007 18399 catalog_manager.cc:5495] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7b3217648914448b7eaa923a4e7d3d3 (127.2.243.131). New cstate: current_term: 1 leader_uuid: "d7b3217648914448b7eaa923a4e7d3d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } health_report { overall_health: UNKNOWN } } }
I20260620 11:36:09.011034 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:09.008868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52000 (local address 127.2.243.131:45031)
0620 11:36:09.008905 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:09.008907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:09.008939 (+ 32us) server_negotiation.cc:407] Connection header received
0620 11:36:09.009006 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:09.009007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:09.009021 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:09.009084 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:36:09.009176 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:09.009615 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:09.010335 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:09.010512 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:09.010564 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:09.010722 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:36:09.010807 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:36:09.010910 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:09.010960 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 11:36:09.010985 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:36:09.012873 18678 raft_consensus.cc:1261] T a47127de9b3a490da12b201b2344cf90 P ba680ca73e064a0cbde4f12e7c2c4080 [term 1 FOLLOWER]: Refusing update from remote peer d7b3217648914448b7eaa923a4e7d3d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:36:09.012873 18544 raft_consensus.cc:1261] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 1 FOLLOWER]: Refusing update from remote peer d7b3217648914448b7eaa923a4e7d3d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:36:09.013202 18881 consensus_queue.cc:1035] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:36:09.013346 18885 consensus_queue.cc:1035] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:36:09.015671 18888 mvcc.cc:205] Tried to move back new op lower bound from 7298889191473774592 to 7298889191441838080. Current Snapshot: MvccSnapshot[applied={T|T < 7298889191473774592}]
I20260620 11:36:09.016559 18890 mvcc.cc:205] Tried to move back new op lower bound from 7298889191473774592 to 7298889191441838080. Current Snapshot: MvccSnapshot[applied={T|T < 7298889191473774592}]
I20260620 11:36:09.017769 18398 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:36:09.017866 18398 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:36:09.018621 18398 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0: Sending DeleteTablet for 3 replicas of tablet a47127de9b3a490da12b201b2344cf90
I20260620 11:36:09.018950 18658 tablet_service.cc:1514] Processing DeleteTablet for tablet a47127de9b3a490da12b201b2344cf90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:36:09 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:53738
I20260620 11:36:09.018949 18524 tablet_service.cc:1514] Processing DeleteTablet for tablet a47127de9b3a490da12b201b2344cf90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:36:09 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:45564
I20260620 11:36:09.018978 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 18461
I20260620 11:36:09.019462 18792 tablet_service.cc:1514] Processing DeleteTablet for tablet a47127de9b3a490da12b201b2344cf90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:36:09 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:51996
I20260620 11:36:09.019701 18899 tablet_replica.cc:330] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3: stopping tablet replica
I20260620 11:36:09.019806 18899 raft_consensus.cc:2229] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:36:09.019898 18899 raft_consensus.cc:2258] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:36:09.020284 18899 ts_tablet_manager.cc:1905] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:36:09.021958 18899 ts_tablet_manager.cc:1918] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:36:09.022037 18899 log.cc:1196] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/a47127de9b3a490da12b201b2344cf90
I20260620 11:36:09.022331 18899 ts_tablet_manager.cc:1939] T a47127de9b3a490da12b201b2344cf90 P d7b3217648914448b7eaa923a4e7d3d3: Deleting consensus metadata
I20260620 11:36:09.022768 18386 catalog_manager.cc:4841] TS d7b3217648914448b7eaa923a4e7d3d3 (127.2.243.131:45031): tablet a47127de9b3a490da12b201b2344cf90 (table test-table [id=0b12d6eb51ac4d9da22e82a54c06aa71]) successfully deleted
I20260620 11:36:09.023049 18898 tablet_replica.cc:330] T a47127de9b3a490da12b201b2344cf90 P ba680ca73e064a0cbde4f12e7c2c4080: stopping tablet replica
I20260620 11:36:09.023181 18898 raft_consensus.cc:2229] T a47127de9b3a490da12b201b2344cf90 P ba680ca73e064a0cbde4f12e7c2c4080 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:36:09.023301 18898 raft_consensus.cc:2258] T a47127de9b3a490da12b201b2344cf90 P ba680ca73e064a0cbde4f12e7c2c4080 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:36:09.023723 18898 ts_tablet_manager.cc:1905] T a47127de9b3a490da12b201b2344cf90 P ba680ca73e064a0cbde4f12e7c2c4080: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:36:09.025318 18898 ts_tablet_manager.cc:1918] T a47127de9b3a490da12b201b2344cf90 P ba680ca73e064a0cbde4f12e7c2c4080: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:36:09.025398 18898 log.cc:1196] T a47127de9b3a490da12b201b2344cf90 P ba680ca73e064a0cbde4f12e7c2c4080: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/wal/wals/a47127de9b3a490da12b201b2344cf90
I20260620 11:36:09.025674 18898 ts_tablet_manager.cc:1939] T a47127de9b3a490da12b201b2344cf90 P ba680ca73e064a0cbde4f12e7c2c4080: Deleting consensus metadata
W20260620 11:36:09.025921 18385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.243.129:32799 (error 108)
I20260620 11:36:09.026095 18383 catalog_manager.cc:4841] TS ba680ca73e064a0cbde4f12e7c2c4080 (127.2.243.130:39745): tablet a47127de9b3a490da12b201b2344cf90 (table test-table [id=0b12d6eb51ac4d9da22e82a54c06aa71]) successfully deleted
I20260620 11:36:09.026175 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 18595
I20260620 11:36:09.027088 18869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:09.026970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:32799 (local address 127.0.0.1:45570)
0620 11:36:09.027017 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:09.027056 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.129:32799: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 11:36:09.027207 18385 catalog_manager.cc:4584] TS 59c49a3f678c4ebdb8aa4063e5f31a2b (127.2.243.129:32799): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a47127de9b3a490da12b201b2344cf90: Network error: Client connection negotiation failed: client connection to 127.2.243.129:32799: connect: Connection refused (error 111)
I20260620 11:36:09.030915 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 18729
I20260620 11:36:09.036274 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 18367
I20260620 11:36:29.041834 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:40113
--webserver_interface=127.2.243.190
--webserver_port=36629
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:40113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:36:29.121819 18901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:36:29.122010 18901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:36:29.122032 18901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:36:29.123204 18901 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 11:36:29.123245 18901 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 11:36:29.123258 18901 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 11:36:29.123270 18901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:36:29.123282 18901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:36:29.123296 18901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:36:29.123347 18901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:36:29.123364 18901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:36:29.124809 18901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:40113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:40113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=36629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:36:29.125056 18901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:36:29.125277 18901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:36:29.133719 18907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:36:29.133728 18908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:36:29.133893 18901 server_base.cc:1034] running on GCE node
W20260620 11:36:29.133719 18910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:36:29.134593 18901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:36:29.134994 18901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:36:29.136143 18901 hybrid_clock.cc:648] HybridClock initialized: now 1781955389136125 us; error 36 us; skew 500 ppm
Jun 20 11:36:29 dist-test-slave-ctbw krb5kdc[18354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955389, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:36:29.143244 18901 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:36:29.143782 18901 webserver.cc:460] Webserver started at http://127.2.243.190:36629/ using document root <none> and password file <none>
I20260620 11:36:29.144033 18901 fs_manager.cc:359] Metadata directory not provided
I20260620 11:36:29.144124 18901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:36:29.145519 18901 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:36:29.146229 18917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:29.146391 18901 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:36:29.146502 18901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "85eba415a72a4fc7ad11ac4d98270ae0"
format_stamp: "Formatted at 2026-06-20 11:36:08 on dist-test-slave-ctbw"
server_key: "a713d011fee0dbc9199d4e2760253adc"
server_key_iv: "53060bef3130c5080ed92b884ebbf9ab"
server_key_version: "encryptionkey@0"
I20260620 11:36:29.146983 18901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:36:29.168077 18901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:36:29.170169 18901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:36:29.170331 18901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:36:29.174564 18901 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:40113
I20260620 11:36:29.174625 18969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:40113 every 8 connection(s)
I20260620 11:36:29.175040 18901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:36:29.176461 18970 sys_catalog.cc:263] Verifying existing consensus state
I20260620 11:36:29.176995 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 18901
I20260620 11:36:29.177307 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:32799
--local_ip_for_outbound_sockets=127.2.243.129
--tserver_master_addrs=127.2.243.190:40113
--webserver_port=34769
--webserver_interface=127.2.243.129
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:36:29.177315 18970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0: Bootstrap starting.
I20260620 11:36:29.185933 18970 log.cc:824] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0: Log is configured to *not* fsync() on all Append() calls
I20260620 11:36:29.187917 18970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:36:29.188203 18970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0: Bootstrap complete.
I20260620 11:36:29.190341 18970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40113 } }
I20260620 11:36:29.190550 18970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85eba415a72a4fc7ad11ac4d98270ae0, State: Initialized, Role: FOLLOWER
I20260620 11:36:29.190759 18970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40113 } }
I20260620 11:36:29.190821 18970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:36:29.190896 18970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:36:29.190946 18970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 1 FOLLOWER]: Advancing to term 2
I20260620 11:36:29.191896 18970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40113 } }
I20260620 11:36:29.191979 18970 leader_election.cc:302] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85eba415a72a4fc7ad11ac4d98270ae0; no voters:
I20260620 11:36:29.192174 18970 leader_election.cc:288] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 11:36:29.192230 18974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 11:36:29.192374 18974 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 2 LEADER]: Becoming Leader. State: Replica: 85eba415a72a4fc7ad11ac4d98270ae0, State: Running, Role: LEADER
I20260620 11:36:29.192440 18970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:36:29.192466 18974 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40113 } }
I20260620 11:36:29.192682 18974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85eba415a72a4fc7ad11ac4d98270ae0. Latest consensus state: current_term: 2 leader_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40113 } } }
I20260620 11:36:29.192762 18974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [sys.catalog]: This master's current role is: LEADER
I20260620 11:36:29.192685 18975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40113 } } }
I20260620 11:36:29.192920 18975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [sys.catalog]: This master's current role is: LEADER
I20260620 11:36:29.193235 18978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:36:29.193993 18978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:36:29.194093 18978 catalog_manager.cc:1259] Loaded cluster ID: 1ed260a67ba7455e87e8c8e63f440937
I20260620 11:36:29.194360 18978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:36:29.195549 18978 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:36:29.195725 18978 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0: Loaded TSK: 0
I20260620 11:36:29.208829 18978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0: Generated new TSK 1
I20260620 11:36:29.208918 18978 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 11:36:29.259131 18972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:36:29.259343 18972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:36:29.259383 18972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:36:29.260515 18972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:36:29.260586 18972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:36:29.260634 18972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:36:29.262121 18972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:32799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=34769
--tserver_master_addrs=127.2.243.190:40113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:36:29.262359 18972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:36:29.262581 18972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:36:29.267517 18997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:36:29.267530 18998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:36:29.267560 19000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:36:29.268038 18972 server_base.cc:1034] running on GCE node
I20260620 11:36:29.268263 18972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:36:29.268512 18972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:36:29.269658 18972 hybrid_clock.cc:648] HybridClock initialized: now 1781955389269639 us; error 37 us; skew 500 ppm
Jun 20 11:36:29 dist-test-slave-ctbw krb5kdc[18354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955389, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:36:29.272300 18972 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:36:29.272753 18972 webserver.cc:460] Webserver started at http://127.2.243.129:34769/ using document root <none> and password file <none>
I20260620 11:36:29.272967 18972 fs_manager.cc:359] Metadata directory not provided
I20260620 11:36:29.273042 18972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:36:29.274432 18972 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:36:29.275194 19007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:29.275370 18972 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:36:29.275470 18972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b"
format_stamp: "Formatted at 2026-06-20 11:36:08 on dist-test-slave-ctbw"
server_key: "d6b148c71b660d8814cf3879ce5ad72e"
server_key_iv: "903d8eec430bd277016f1a3a5184537d"
server_key_version: "encryptionkey@0"
I20260620 11:36:29.275810 18972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:36:29.295615 18972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:36:29.297770 18972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:36:29.297935 18972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:36:29.298177 18972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:36:29.298632 19014 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 11:36:29.300689 18972 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 11:36:29.300745 18972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 11:36:29.300789 18972 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 11:36:29.301466 18972 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 11:36:29.301520 18972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 11:36:29.301589 19014 tablet_bootstrap.cc:492] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Bootstrap starting.
I20260620 11:36:29.308688 18972 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:32799
I20260620 11:36:29.309131 18972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:36:29.312386 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 18972
I20260620 11:36:29.312481 19014 log.cc:824] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Log is configured to *not* fsync() on all Append() calls
I20260620 11:36:29.312692 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:39745
--local_ip_for_outbound_sockets=127.2.243.130
--tserver_master_addrs=127.2.243.190:40113
--webserver_port=36341
--webserver_interface=127.2.243.130
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 11:36:29 dist-test-slave-ctbw krb5kdc[18354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955389, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:36:29.318977 19121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:32799 every 8 connection(s)
I20260620 11:36:29.321493 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:29.309879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:42335 (local address 127.2.243.190:40113)
0620 11:36:29.311006 (+ 1127us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:29.311012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:29.311039 (+ 27us) server_negotiation.cc:407] Connection header received
0620 11:36:29.311158 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:29.311162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:29.311202 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:29.311337 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:36:29.311983 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.313397 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.314926 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.315387 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.315525 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:29.317719 (+ 2194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:36:29.317722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:36:29.317722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:36:29.317733 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:36:29.319702 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:29.320130 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:29.320132 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:29.320133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:29.320225 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:29.320538 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:29.320538 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:29.320539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:29.320789 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:36:29.320875 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:29.321163 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 11:36:29.321285 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1075,"thread_start_us":98,"threads_started":1}
I20260620 11:36:29.321493 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:29.309802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.129:42335)
0620 11:36:29.309980 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:29.310020 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 11:36:29.310497 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:36:29.311366 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:36:29.311370 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:36:29.311678 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:36:29.311864 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:29.311869 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.313541 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.313543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:29.314793 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:29.314796 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.315514 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.315515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:29.315643 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:29.316179 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:36:29.316187 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:36:29.317606 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:36:29.319797 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:36:29.319804 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:36:29.319806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:36:29.320028 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:36:29.320306 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:36:29.320307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:36:29.320308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:36:29.320416 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:36:29.320868 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:36:29.320874 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:36:29.321004 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 11:36:29.321112 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 11:36:29.321286 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":109,"threads_started":1}
I20260620 11:36:29.321487 19014 tablet_bootstrap.cc:492] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:36:29.321852 19014 tablet_bootstrap.cc:492] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Bootstrap complete.
I20260620 11:36:29.322621 19122 heartbeater.cc:347] Connected to a master server at 127.2.243.190:40113
I20260620 11:36:29.322728 19122 heartbeater.cc:464] Registering TS with master...
I20260620 11:36:29.322949 19122 heartbeater.cc:511] Master 127.2.243.190:40113 requested a full tablet report, sending...
I20260620 11:36:29.323057 19014 ts_tablet_manager.cc:1397] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.009s
I20260620 11:36:29.323742 18934 ts_manager.cc:194] Registered new tserver with Master: 59c49a3f678c4ebdb8aa4063e5f31a2b (127.2.243.129:32799)
I20260620 11:36:29.324219 19014 raft_consensus.cc:348] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:29.324520 19014 raft_consensus.cc:729] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59c49a3f678c4ebdb8aa4063e5f31a2b, State: Initialized, Role: FOLLOWER
I20260620 11:36:29.324638 19014 consensus_queue.cc:260] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:29.324882 19014 ts_tablet_manager.cc:1428] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 11:36:29.324985 18934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:42335
I20260620 11:36:29.326038 19122 heartbeater.cc:503] Master 127.2.243.190:40113 was elected leader, sending a full tablet report...
I20260620 11:36:29.330672 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:29.325439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51662 (local address 127.2.243.129:32799)
0620 11:36:29.325579 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:29.325584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:29.326485 (+ 901us) server_negotiation.cc:407] Connection header received
0620 11:36:29.326519 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:29.326522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:29.326539 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:29.326601 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:36:29.326812 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.327987 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.329493 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.330230 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.330303 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:29.330354 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:29.330564 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 11:36:29.330602 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":66,"threads_started":1}
I20260620 11:36:29.330680 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:29.325242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:32799 (local address 127.0.0.1:51662)
0620 11:36:29.326319 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:29.326354 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 11:36:29.326412 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:36:29.326652 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:36:29.326655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:36:29.326657 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:36:29.326730 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:29.326731 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.328086 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.328087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:29.329412 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:29.329416 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.330319 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.330320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:29.330435 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:29.330452 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 11:36:29.330542 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 11:36:29.330609 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1049,"thread_start_us":90,"threads_started":1}
I20260620 11:36:29.331293 19056 tablet_service.cc:1514] Processing DeleteTablet for tablet a47127de9b3a490da12b201b2344cf90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:36:09 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:51662
I20260620 11:36:29.331843 19131 tablet_replica.cc:330] stopping tablet replica
I20260620 11:36:29.331943 19131 raft_consensus.cc:2229] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:36:29.332007 19131 raft_consensus.cc:2258] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:36:29.332304 19131 ts_tablet_manager.cc:1905] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:36:29.334455 19131 ts_tablet_manager.cc:1918] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:36:29.334523 19131 log.cc:1196] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/wal/wals/a47127de9b3a490da12b201b2344cf90
I20260620 11:36:29.334743 19131 ts_tablet_manager.cc:1939] T a47127de9b3a490da12b201b2344cf90 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Deleting consensus metadata
I20260620 11:36:29.335352 18920 catalog_manager.cc:4841] TS 59c49a3f678c4ebdb8aa4063e5f31a2b (127.2.243.129:32799): tablet a47127de9b3a490da12b201b2344cf90 (table test-table [id=0b12d6eb51ac4d9da22e82a54c06aa71]) successfully deleted
W20260620 11:36:29.402029 19127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:36:29.402283 19127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:36:29.402329 19127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:36:29.403513 19127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:36:29.403581 19127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:36:29.403628 19127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:36:29.405066 19127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:39745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=36341
--tserver_master_addrs=127.2.243.190:40113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:36:29.405354 19127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:36:29.405596 19127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:36:29.410595 19136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:36:29.410604 19139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:36:29.410629 19137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:36:29.410912 19127 server_base.cc:1034] running on GCE node
I20260620 11:36:29.411177 19127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:36:29.411373 19127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:36:29.412547 19127 hybrid_clock.cc:648] HybridClock initialized: now 1781955389412532 us; error 32 us; skew 500 ppm
Jun 20 11:36:29 dist-test-slave-ctbw krb5kdc[18354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955389, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:36:29.415889 19127 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:36:29.416484 19127 webserver.cc:460] Webserver started at http://127.2.243.130:36341/ using document root <none> and password file <none>
I20260620 11:36:29.416738 19127 fs_manager.cc:359] Metadata directory not provided
I20260620 11:36:29.416814 19127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:36:29.418169 19127 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:36:29.418756 19146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:29.418946 19127 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:36:29.419049 19127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "ba680ca73e064a0cbde4f12e7c2c4080"
format_stamp: "Formatted at 2026-06-20 11:36:08 on dist-test-slave-ctbw"
server_key: "e086db3aed0c76b25b9b7115a468500d"
server_key_iv: "cf2d1f3b7efb8c6d5bb68ba680e5b5dc"
server_key_version: "encryptionkey@0"
I20260620 11:36:29.419397 19127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:36:29.432992 19127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:36:29.435273 19127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:36:29.435420 19127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:36:29.435636 19127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:36:29.435926 19127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:36:29.435974 19127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:29.436015 19127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:36:29.436043 19127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:29.442034 19127 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:39745
I20260620 11:36:29.442101 19259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:39745 every 8 connection(s)
I20260620 11:36:29.442409 19127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
Jun 20 11:36:29 dist-test-slave-ctbw krb5kdc[18354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955389, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:36:29.448360 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 19127
I20260620 11:36:29.448654 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:45031
--local_ip_for_outbound_sockets=127.2.243.131
--tserver_master_addrs=127.2.243.190:40113
--webserver_port=35113
--webserver_interface=127.2.243.131
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:36:29.451776 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:29.443090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:60315 (local address 127.2.243.190:40113)
0620 11:36:29.443130 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:29.443132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:29.443327 (+ 195us) server_negotiation.cc:407] Connection header received
0620 11:36:29.443811 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:29.443813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:29.443829 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:29.443906 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:36:29.444829 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.445322 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.446250 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.446443 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.446502 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:29.448693 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:36:29.448695 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:36:29.448695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:36:29.448703 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:36:29.450386 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:29.450771 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:29.450772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:29.450772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:29.450836 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:29.451112 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:29.451113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:29.451113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:29.451245 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:36:29.451331 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:29.451636 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 11:36:29.451711 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 11:36:29.452062 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:29.443077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.130:60315)
0620 11:36:29.443202 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:29.443236 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 11:36:29.443698 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:36:29.443953 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:36:29.443956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:36:29.444511 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:36:29.444732 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:29.444737 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.445399 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.445400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:29.446175 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:29.446177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.446532 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.446533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:29.446634 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:29.447193 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:36:29.447201 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:36:29.448596 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:36:29.450463 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:36:29.450468 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:36:29.450470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:36:29.450697 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:36:29.450932 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:36:29.450933 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:36:29.450933 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:36:29.451048 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:36:29.451328 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:36:29.451335 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:36:29.451496 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 11:36:29.451606 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 11:36:29.451776 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":67,"threads_started":1}
I20260620 11:36:29.452595 19260 heartbeater.cc:347] Connected to a master server at 127.2.243.190:40113
I20260620 11:36:29.452708 19260 heartbeater.cc:464] Registering TS with master...
I20260620 11:36:29.452881 19260 heartbeater.cc:511] Master 127.2.243.190:40113 requested a full tablet report, sending...
I20260620 11:36:29.453274 18934 ts_manager.cc:194] Registered new tserver with Master: ba680ca73e064a0cbde4f12e7c2c4080 (127.2.243.130:39745)
I20260620 11:36:29.453653 18934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:60315
W20260620 11:36:29.530567 19263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:36:29.530755 19263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:36:29.530778 19263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:36:29.532183 19263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:36:29.532243 19263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:36:29.532274 19263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:36:29.533793 19263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:45031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=35113
--tserver_master_addrs=127.2.243.190:40113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:36:29.534049 19263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:36:29.534286 19263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:36:29.539296 19271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:36:29.539290 19268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:36:29.539410 19263 server_base.cc:1034] running on GCE node
W20260620 11:36:29.539290 19269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:36:29.539731 19263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:36:29.539997 19263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:36:29.541167 19263 hybrid_clock.cc:648] HybridClock initialized: now 1781955389541138 us; error 50 us; skew 500 ppm
Jun 20 11:36:29 dist-test-slave-ctbw krb5kdc[18354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955389, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:36:29.543628 19263 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:36:29.544142 19263 webserver.cc:460] Webserver started at http://127.2.243.131:35113/ using document root <none> and password file <none>
I20260620 11:36:29.544382 19263 fs_manager.cc:359] Metadata directory not provided
I20260620 11:36:29.544461 19263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:36:29.545779 19263 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:36:29.546360 19278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:29.546523 19263 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:36:29.546617 19263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "d7b3217648914448b7eaa923a4e7d3d3"
format_stamp: "Formatted at 2026-06-20 11:36:08 on dist-test-slave-ctbw"
server_key: "2731c599644d514da4476b31863bb525"
server_key_iv: "4bc261953bb41bcf6ded55b304ee65d4"
server_key_version: "encryptionkey@0"
I20260620 11:36:29.546998 19263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:36:29.553705 19263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:36:29.555548 19263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:36:29.555693 19263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:36:29.555902 19263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:36:29.556188 19263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:36:29.556238 19263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:29.556278 19263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:36:29.556306 19263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:29.561949 19263 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:45031
I20260620 11:36:29.562008 19391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:45031 every 8 connection(s)
I20260620 11:36:29.562359 19263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:36:29.564261 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 19263
Jun 20 11:36:29 dist-test-slave-ctbw krb5kdc[18354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955389, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:36:29.571832 19396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:29.564698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36590 (local address 127.2.243.190:40113)
0620 11:36:29.564844 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:29.564847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:29.565499 (+ 652us) server_negotiation.cc:407] Connection header received
0620 11:36:29.565514 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:29.565515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:29.565523 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:29.565590 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:36:29.566149 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.566623 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.567449 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.567662 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.567743 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:29.569016 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:36:29.569017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:36:29.569017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:36:29.569025 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:36:29.570568 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:29.570991 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:29.570992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:29.570992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:29.571053 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:29.571243 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:29.571244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:29.571244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:29.571355 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:36:29.571539 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:29.571577 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 11:36:29.571669 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":62,"threads_started":1}
I20260620 11:36:29.571844 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:29.563073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:59459 (local address 127.2.243.190:40113)
0620 11:36:29.563122 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:29.563124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:29.563370 (+ 246us) server_negotiation.cc:407] Connection header received
0620 11:36:29.563857 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:29.563857 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:29.563867 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:29.563918 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:36:29.564923 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.565435 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.566391 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.566598 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.566679 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:29.568826 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:36:29.568827 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:36:29.568828 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:36:29.568835 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:36:29.570432 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:29.570828 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:29.570829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:29.570829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:29.570904 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:29.571140 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:29.571142 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:29.571142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:29.571254 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:36:29.571327 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:29.571585 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 11:36:29.571725 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":18}
I20260620 11:36:29.572002 19394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:29.563089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.131:59459)
0620 11:36:29.563241 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:29.563274 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 11:36:29.563774 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:36:29.563972 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:36:29.563975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:36:29.564584 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:36:29.564841 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:29.564846 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.565510 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.565511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:29.566281 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:29.566282 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.566689 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.566690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:29.566820 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:29.567390 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:36:29.567399 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:36:29.568729 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:36:29.570528 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:36:29.570535 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:36:29.570536 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:36:29.570747 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:36:29.570968 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:36:29.570969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:36:29.570969 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:36:29.571073 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:36:29.571317 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:36:29.571323 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:36:29.571460 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 11:36:29.571565 (+ 105us) client_negotiation.cc:240] Negotiation successful
0620 11:36:29.571723 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":82,"threads_started":1}
I20260620 11:36:29.572404 19392 heartbeater.cc:347] Connected to a master server at 127.2.243.190:40113
I20260620 11:36:29.572515 19392 heartbeater.cc:464] Registering TS with master...
I20260620 11:36:29.572695 19392 heartbeater.cc:511] Master 127.2.243.190:40113 requested a full tablet report, sending...
I20260620 11:36:29.573043 18934 ts_manager.cc:194] Registered new tserver with Master: d7b3217648914448b7eaa923a4e7d3d3 (127.2.243.131:45031)
I20260620 11:36:29.573400 18934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:59459
I20260620 11:36:29.583416 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:36:29.585937 19396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:29.583763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36604 (local address 127.2.243.190:40113)
0620 11:36:29.583815 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:29.583817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:29.583900 (+ 83us) server_negotiation.cc:407] Connection header received
0620 11:36:29.583961 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:29.583961 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:29.583991 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:29.584050 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:36:29.584163 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.584657 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.585363 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.585567 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.585616 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:29.585730 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:36:29.585770 (+ 40us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 11:36:29.585890 (+ 120us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36604: token expired
Metrics: {"server-negotiator.queue_time_us":31}
W20260620 11:36:29.586004 19396 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36604: token expired
W20260620 11:36:29.586195 18864 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.2.243.190:40113)
I20260620 11:36:29.594224 19396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:29.586452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.2.243.190:40113)
0620 11:36:29.586488 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:29.586490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:29.586589 (+ 99us) server_negotiation.cc:407] Connection header received
0620 11:36:29.586636 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:29.586636 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:29.586647 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:29.586715 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:36:29.587466 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.588238 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.589309 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.589658 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.589741 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:29.591310 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:36:29.591311 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:36:29.591312 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:36:29.591320 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:36:29.592982 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:29.593337 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:29.593339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:29.593339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:29.593393 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:29.593679 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:29.593680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:29.593680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:29.593887 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:36:29.593979 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:29.594058 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 11:36:29.594132 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 11:36:29.595355 18934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36612:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:36:29.597052 18934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:36:29.599216 19056 tablet_service.cc:1467] Processing CreateTablet for tablet b166b43f77a24dd3900977a8833fe962 (DEFAULT_TABLE table=test-table [id=bb7b3df4075d49749ac56e3bcf6e26a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:36:29.599427 19056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b166b43f77a24dd3900977a8833fe962. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:36:29.600926 19014 tablet_bootstrap.cc:492] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Bootstrap starting.
I20260620 11:36:29.601236 19014 tablet_bootstrap.cc:654] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Neither blocks nor log segments found. Creating new log.
I20260620 11:36:29.601816 19014 tablet_bootstrap.cc:492] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b: No bootstrap required, opened a new log
I20260620 11:36:29.601873 19014 ts_tablet_manager.cc:1397] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 11:36:29.601992 19014 raft_consensus.cc:348] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:29.602068 19014 raft_consensus.cc:374] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:36:29.602118 19014 raft_consensus.cc:729] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59c49a3f678c4ebdb8aa4063e5f31a2b, State: Initialized, Role: FOLLOWER
I20260620 11:36:29.602159 19014 consensus_queue.cc:260] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:29.602241 19014 ts_tablet_manager.cc:1428] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:29.603104 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:29.598842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:39745 (local address 127.0.0.1:36786)
0620 11:36:29.599001 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:29.599010 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:36:29.599047 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:36:29.599443 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:36:29.599444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:36:29.599446 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:36:29.599501 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:29.599503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.600668 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.600669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:29.601907 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:29.601910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.602839 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.602840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:29.602920 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:29.602924 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:36:29.602985 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 11:36:29.603034 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":105,"threads_started":1}
I20260620 11:36:29.603116 19401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:29.599090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36786 (local address 127.2.243.130:39745)
0620 11:36:29.599283 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:29.599288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:29.599300 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:36:29.599342 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:29.599344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:29.599360 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:29.599448 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:36:29.599571 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.600584 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.602028 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.602742 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.602836 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:29.602929 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:29.602999 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 11:36:29.603037 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
I20260620 11:36:29.603859 19194 tablet_service.cc:1467] Processing CreateTablet for tablet b166b43f77a24dd3900977a8833fe962 (DEFAULT_TABLE table=test-table [id=bb7b3df4075d49749ac56e3bcf6e26a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:36:29.604216 19194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b166b43f77a24dd3900977a8833fe962. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:36:29.604427 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:29.598785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:45031 (local address 127.0.0.1:57678)
0620 11:36:29.598822 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:29.598833 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:36:29.598902 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:36:29.601181 (+ 2279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:36:29.601182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:36:29.601184 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:36:29.601223 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:29.601224 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.602461 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.602462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:29.603513 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:29.603515 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.604219 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.604220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:29.604281 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:29.604284 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:36:29.604339 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 11:36:29.604382 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260620 11:36:29.604481 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:29.598894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.2.243.131:45031)
0620 11:36:29.601014 (+ 2120us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:29.601019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:29.601029 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:36:29.601072 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:29.601074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:29.601090 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:29.601184 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:36:29.601275 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.602229 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.603591 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.604116 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.604205 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:29.604260 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:29.604358 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 11:36:29.604405 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2066,"thread_start_us":114,"threads_started":1}
I20260620 11:36:29.605296 19326 tablet_service.cc:1467] Processing CreateTablet for tablet b166b43f77a24dd3900977a8833fe962 (DEFAULT_TABLE table=test-table [id=bb7b3df4075d49749ac56e3bcf6e26a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:36:29.605602 19326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b166b43f77a24dd3900977a8833fe962. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:36:29.607717 19403 tablet_bootstrap.cc:492] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080: Bootstrap starting.
I20260620 11:36:29.608282 19403 tablet_bootstrap.cc:654] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080: Neither blocks nor log segments found. Creating new log.
I20260620 11:36:29.608809 19404 tablet_bootstrap.cc:492] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3: Bootstrap starting.
I20260620 11:36:29.608809 19403 log.cc:824] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080: Log is configured to *not* fsync() on all Append() calls
I20260620 11:36:29.609418 19404 tablet_bootstrap.cc:654] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3: Neither blocks nor log segments found. Creating new log.
I20260620 11:36:29.609525 19403 tablet_bootstrap.cc:492] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080: No bootstrap required, opened a new log
I20260620 11:36:29.609599 19403 ts_tablet_manager.cc:1397] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:36:29.610011 19404 log.cc:824] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3: Log is configured to *not* fsync() on all Append() calls
I20260620 11:36:29.610700 19404 tablet_bootstrap.cc:492] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3: No bootstrap required, opened a new log
I20260620 11:36:29.610750 19404 ts_tablet_manager.cc:1397] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:36:29.611424 19403 raft_consensus.cc:348] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:29.611581 19403 raft_consensus.cc:374] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:36:29.611626 19403 raft_consensus.cc:729] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba680ca73e064a0cbde4f12e7c2c4080, State: Initialized, Role: FOLLOWER
I20260620 11:36:29.611743 19403 consensus_queue.cc:260] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:29.612044 19260 heartbeater.cc:503] Master 127.2.243.190:40113 was elected leader, sending a full tablet report...
I20260620 11:36:29.612030 19403 ts_tablet_manager.cc:1428] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 11:36:29.612332 19404 raft_consensus.cc:348] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:29.612522 19404 raft_consensus.cc:374] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:36:29.612571 19404 raft_consensus.cc:729] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b3217648914448b7eaa923a4e7d3d3, State: Initialized, Role: FOLLOWER
I20260620 11:36:29.612679 19404 consensus_queue.cc:260] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:29.612950 19404 ts_tablet_manager.cc:1428] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:36:29.612965 19392 heartbeater.cc:503] Master 127.2.243.190:40113 was elected leader, sending a full tablet report...
I20260620 11:36:29.666738 19128 raft_consensus.cc:482] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:36:29.666929 19128 raft_consensus.cc:504] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:29.667397 19128 leader_election.cc:288] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7b3217648914448b7eaa923a4e7d3d3 (127.2.243.131:45031), ba680ca73e064a0cbde4f12e7c2c4080 (127.2.243.130:39745)
I20260620 11:36:29.670177 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:29.667582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:53029 (local address 127.2.243.131:45031)
0620 11:36:29.667644 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:29.667646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:29.667654 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:36:29.667727 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:29.667729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:29.667739 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:29.667800 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:36:29.667980 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.668539 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.669469 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.669833 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.669912 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:29.669921 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:29.670076 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 11:36:29.670109 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 11:36:29.670192 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:29.667507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:45031 (local address 127.2.243.129:53029)
0620 11:36:29.667573 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:29.667587 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 11:36:29.667647 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:36:29.667814 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:36:29.667816 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:36:29.667817 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:36:29.667897 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:29.667899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.668616 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.668617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:29.669364 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:29.669366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.669911 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.669912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:29.669993 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:29.669996 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:36:29.670069 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 11:36:29.670107 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 11:36:29.670476 19401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:29.667752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:51397 (local address 127.2.243.130:39745)
0620 11:36:29.667836 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:29.667838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:29.667851 (+ 13us) server_negotiation.cc:407] Connection header received
0620 11:36:29.667925 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:29.667926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:29.667936 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:29.668014 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:36:29.668180 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.668723 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.669665 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.670069 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.670161 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:29.670171 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:29.670360 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 11:36:29.670410 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 11:36:29.670579 19346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b166b43f77a24dd3900977a8833fe962" candidate_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b3217648914448b7eaa923a4e7d3d3" is_pre_election: true
I20260620 11:36:29.670717 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:29.667572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:39745 (local address 127.2.243.129:51397)
0620 11:36:29.667791 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:29.667802 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:36:29.667861 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:36:29.668016 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:36:29.668017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:36:29.668018 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:36:29.668106 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:29.668108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.668830 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.668831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:29.669567 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:29.669569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.670179 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.670180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:29.670252 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:29.670256 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:36:29.670568 (+ 312us) client_negotiation.cc:240] Negotiation successful
0620 11:36:29.670611 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":113,"threads_started":1}
I20260620 11:36:29.670758 19346 raft_consensus.cc:2454] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59c49a3f678c4ebdb8aa4063e5f31a2b in term 0.
I20260620 11:36:29.671010 19011 leader_election.cc:302] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59c49a3f678c4ebdb8aa4063e5f31a2b, d7b3217648914448b7eaa923a4e7d3d3; no voters:
I20260620 11:36:29.671093 19214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b166b43f77a24dd3900977a8833fe962" candidate_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" is_pre_election: true
I20260620 11:36:29.671157 19128 raft_consensus.cc:2781] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:36:29.671253 19128 raft_consensus.cc:482] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:36:29.671257 19214 raft_consensus.cc:2454] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59c49a3f678c4ebdb8aa4063e5f31a2b in term 0.
I20260620 11:36:29.671293 19128 raft_consensus.cc:3037] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:36:29.672173 19128 raft_consensus.cc:504] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:29.672330 19128 leader_election.cc:288] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [CANDIDATE]: Term 1 election: Requested vote from peers d7b3217648914448b7eaa923a4e7d3d3 (127.2.243.131:45031), ba680ca73e064a0cbde4f12e7c2c4080 (127.2.243.130:39745)
I20260620 11:36:29.672466 19346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b166b43f77a24dd3900977a8833fe962" candidate_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7b3217648914448b7eaa923a4e7d3d3"
I20260620 11:36:29.672480 19214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b166b43f77a24dd3900977a8833fe962" candidate_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba680ca73e064a0cbde4f12e7c2c4080"
I20260620 11:36:29.672557 19214 raft_consensus.cc:3037] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:36:29.672556 19346 raft_consensus.cc:3037] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:36:29.673259 19214 raft_consensus.cc:2454] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59c49a3f678c4ebdb8aa4063e5f31a2b in term 1.
I20260620 11:36:29.673416 19346 raft_consensus.cc:2454] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59c49a3f678c4ebdb8aa4063e5f31a2b in term 1.
I20260620 11:36:29.673462 19008 leader_election.cc:302] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59c49a3f678c4ebdb8aa4063e5f31a2b, ba680ca73e064a0cbde4f12e7c2c4080; no voters:
I20260620 11:36:29.673552 19128 raft_consensus.cc:2781] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:36:29.673738 19128 raft_consensus.cc:686] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 1 LEADER]: Becoming Leader. State: Replica: 59c49a3f678c4ebdb8aa4063e5f31a2b, State: Running, Role: LEADER
I20260620 11:36:29.673806 19128 consensus_queue.cc:237] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:29.674584 18934 catalog_manager.cc:5495] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 59c49a3f678c4ebdb8aa4063e5f31a2b (127.2.243.129). New cstate: current_term: 1 leader_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } health_report { overall_health: HEALTHY } } }
W20260620 11:36:29.692971 19261 tablet.cc:2369] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:36:29.695020 19130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:29.692073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51674 (local address 127.2.243.129:32799)
0620 11:36:29.692141 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:29.692143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:29.692155 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:36:29.692249 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:29.692250 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:29.692271 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:29.692412 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:36:29.692692 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.693387 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.694203 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:29.694428 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:29.694504 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:29.694661 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:36:29.694770 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:36:29.694858 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:29.694913 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 11:36:29.694937 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 11:36:29.696534 19346 raft_consensus.cc:1261] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3 [term 1 FOLLOWER]: Refusing update from remote peer 59c49a3f678c4ebdb8aa4063e5f31a2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:36:29.696534 19214 raft_consensus.cc:1261] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [term 1 FOLLOWER]: Refusing update from remote peer 59c49a3f678c4ebdb8aa4063e5f31a2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:36:29.696857 19128 consensus_queue.cc:1035] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:36:29.697005 19410 consensus_queue.cc:1035] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:36:29.699524 19415 mvcc.cc:205] Tried to move back new op lower bound from 7298889276195041280 to 7298889276104998912. Current Snapshot: MvccSnapshot[applied={T|T < 7298889276195041280}]
I20260620 11:36:29.701552 18932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:36:29.701653 18932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36612:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:36:29.702344 18932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0: Sending DeleteTablet for 3 replicas of tablet b166b43f77a24dd3900977a8833fe962
I20260620 11:36:29.702690 19194 tablet_service.cc:1514] Processing DeleteTablet for tablet b166b43f77a24dd3900977a8833fe962 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:36:29 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:36786
I20260620 11:36:29.702788 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 18972
I20260620 11:36:29.702775 19326 tablet_service.cc:1514] Processing DeleteTablet for tablet b166b43f77a24dd3900977a8833fe962 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:36:29 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:57678
I20260620 11:36:29.703570 19424 tablet_replica.cc:330] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3: stopping tablet replica
I20260620 11:36:29.703675 19424 raft_consensus.cc:2229] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:36:29.703778 19424 raft_consensus.cc:2258] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:36:29.704185 19424 ts_tablet_manager.cc:1905] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:36:29.706105 19424 ts_tablet_manager.cc:1918] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:36:29.706182 19424 log.cc:1196] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/b166b43f77a24dd3900977a8833fe962
I20260620 11:36:29.706465 19424 ts_tablet_manager.cc:1939] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3: Deleting consensus metadata
I20260620 11:36:29.706911 18921 catalog_manager.cc:4841] TS d7b3217648914448b7eaa923a4e7d3d3 (127.2.243.131:45031): tablet b166b43f77a24dd3900977a8833fe962 (table test-table [id=bb7b3df4075d49749ac56e3bcf6e26a0]) successfully deleted
I20260620 11:36:29.709295 19423 tablet_replica.cc:330] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080: stopping tablet replica
W20260620 11:36:29.709311 18920 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.243.129:32799 (error 108)
I20260620 11:36:29.709407 19423 raft_consensus.cc:2229] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:36:29.709539 19423 raft_consensus.cc:2258] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 11:36:29.709772 19129 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:36:29.709625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:32799 (local address 127.0.0.1:51686)
0620 11:36:29.709681 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:29.709740 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.129:32799: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 11:36:29.709957 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 19127
W20260620 11:36:29.709889 18920 catalog_manager.cc:4584] TS 59c49a3f678c4ebdb8aa4063e5f31a2b (127.2.243.129:32799): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b166b43f77a24dd3900977a8833fe962: Network error: Client connection negotiation failed: client connection to 127.2.243.129:32799: connect: Connection reset by peer (error 104)
I20260620 11:36:29.714555 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 19263
I20260620 11:36:29.714658 19129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:29.714536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:39745 (local address 127.0.0.1:36792)
0620 11:36:29.714588 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:29.714615 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.130:39745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 11:36:29.719111 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 18901
I20260620 11:36:49.724797 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:40113
--webserver_interface=127.2.243.190
--webserver_port=36629
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:40113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:36:49.805249 19426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:36:49.805442 19426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:36:49.805464 19426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:36:49.806528 19426 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 11:36:49.806560 19426 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 11:36:49.806573 19426 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 11:36:49.806584 19426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:36:49.806597 19426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:36:49.806610 19426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:36:49.806622 19426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:36:49.806634 19426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:36:49.808805 19426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:40113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:40113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=36629
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:36:49.808995 19426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:36:49.809166 19426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:36:49.813990 19431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:36:49.813998 19432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:36:49.813998 19434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:36:49.814265 19426 server_base.cc:1034] running on GCE node
I20260620 11:36:49.814553 19426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:36:49.814841 19426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:36:49.816025 19426 hybrid_clock.cc:648] HybridClock initialized: now 1781955409816008 us; error 39 us; skew 500 ppm
Jun 20 11:36:49 dist-test-slave-ctbw krb5kdc[18354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955409, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:36:49.821565 19426 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:36:49.822018 19426 webserver.cc:460] Webserver started at http://127.2.243.190:36629/ using document root <none> and password file <none>
I20260620 11:36:49.822263 19426 fs_manager.cc:359] Metadata directory not provided
I20260620 11:36:49.822309 19426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:36:49.823731 19426 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:36:49.824419 19441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:49.824640 19426 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:36:49.824713 19426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "85eba415a72a4fc7ad11ac4d98270ae0"
format_stamp: "Formatted at 2026-06-20 11:36:08 on dist-test-slave-ctbw"
server_key: "a713d011fee0dbc9199d4e2760253adc"
server_key_iv: "53060bef3130c5080ed92b884ebbf9ab"
server_key_version: "encryptionkey@0"
I20260620 11:36:49.825032 19426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:36:49.838594 19426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:36:49.840973 19426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:36:49.841087 19426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:36:49.844910 19426 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:40113
I20260620 11:36:49.844983 19493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:40113 every 8 connection(s)
I20260620 11:36:49.845260 19426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:36:49.846707 19494 sys_catalog.cc:263] Verifying existing consensus state
I20260620 11:36:49.847453 19494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0: Bootstrap starting.
I20260620 11:36:49.849879 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 19426
I20260620 11:36:49.850201 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:32799
--local_ip_for_outbound_sockets=127.2.243.129
--tserver_master_addrs=127.2.243.190:40113
--webserver_port=34769
--webserver_interface=127.2.243.129
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:36:49.856601 19494 log.cc:824] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0: Log is configured to *not* fsync() on all Append() calls
I20260620 11:36:49.858839 19494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:36:49.859200 19494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0: Bootstrap complete.
I20260620 11:36:49.861394 19494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40113 } }
I20260620 11:36:49.861642 19494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85eba415a72a4fc7ad11ac4d98270ae0, State: Initialized, Role: FOLLOWER
I20260620 11:36:49.861824 19494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40113 } }
I20260620 11:36:49.861903 19494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:36:49.861975 19494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:36:49.862027 19494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 2 FOLLOWER]: Advancing to term 3
I20260620 11:36:49.863091 19494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40113 } }
I20260620 11:36:49.863210 19494 leader_election.cc:302] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85eba415a72a4fc7ad11ac4d98270ae0; no voters:
I20260620 11:36:49.863407 19494 leader_election.cc:288] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 11:36:49.863451 19498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 11:36:49.863591 19498 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [term 3 LEADER]: Becoming Leader. State: Replica: 85eba415a72a4fc7ad11ac4d98270ae0, State: Running, Role: LEADER
I20260620 11:36:49.863680 19494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:36:49.863696 19498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40113 } }
I20260620 11:36:49.863984 19498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85eba415a72a4fc7ad11ac4d98270ae0. Latest consensus state: current_term: 3 leader_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40113 } } }
I20260620 11:36:49.864061 19498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [sys.catalog]: This master's current role is: LEADER
I20260620 11:36:49.863987 19500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85eba415a72a4fc7ad11ac4d98270ae0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 40113 } } }
I20260620 11:36:49.864223 19500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0 [sys.catalog]: This master's current role is: LEADER
I20260620 11:36:49.864881 19503 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:36:49.865702 19503 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:36:49.865813 19503 catalog_manager.cc:1259] Loaded cluster ID: 1ed260a67ba7455e87e8c8e63f440937
I20260620 11:36:49.865854 19503 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:36:49.866726 19503 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:36:49.866957 19503 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0: Loaded TSK: 0
I20260620 11:36:49.867010 19503 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85eba415a72a4fc7ad11ac4d98270ae0: Loaded TSK: 1
F20260620 11:36:49.867252 19503 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb8580879ad google::LogMessage::Fail() at ??:0
@ 0x7fb85808a88a google::LogMessage::SendToLog() at ??:0
@ 0x7fb8580874de google::LogMessage::Flush() at ??:0
@ 0x7fb858088a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb85a7f4104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb85a7f1d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb85ffe11cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb85ffe148e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb85ffe1594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fb85ffa7c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fb85ffb1ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb859f550d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb859f4a954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb85adae6db start_thread at ??:0
@ 0x7fb856b8561f clone at ??:0
W20260620 11:36:49.966259 19496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:36:49.966543 19496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:36:49.966583 19496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:36:49.968353 19496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:36:49.968418 19496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:36:49.968461 19496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:36:49.970537 19496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:32799
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=34769
--tserver_master_addrs=127.2.243.190:40113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:36:49.970805 19496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:36:49.971117 19496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:36:49.977669 19536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:36:49.977682 19538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:36:49.977804 19535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:36:49.978360 19496 server_base.cc:1034] running on GCE node
I20260620 11:36:49.978586 19496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:36:49.978858 19496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:36:49.980037 19496 hybrid_clock.cc:648] HybridClock initialized: now 1781955409980023 us; error 36 us; skew 500 ppm
Jun 20 11:36:49 dist-test-slave-ctbw krb5kdc[18354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955409, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:36:49.982690 19496 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:36:49.983261 19496 webserver.cc:460] Webserver started at http://127.2.243.129:34769/ using document root <none> and password file <none>
I20260620 11:36:49.983503 19496 fs_manager.cc:359] Metadata directory not provided
I20260620 11:36:49.983572 19496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:36:49.984920 19496 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:36:49.985638 19545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:49.985811 19496 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:36:49.985898 19496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b"
format_stamp: "Formatted at 2026-06-20 11:36:08 on dist-test-slave-ctbw"
server_key: "d6b148c71b660d8814cf3879ce5ad72e"
server_key_iv: "903d8eec430bd277016f1a3a5184537d"
server_key_version: "encryptionkey@0"
I20260620 11:36:49.986260 19496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:36:50.007643 19496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:36:50.010145 19496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:36:50.010319 19496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:36:50.010581 19496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:36:50.011114 19552 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 11:36:50.012593 19496 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 11:36:50.012646 19496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 11:36:50.012670 19496 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 11:36:50.013355 19496 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 11:36:50.013419 19496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 11:36:50.013478 19552 tablet_bootstrap.cc:492] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Bootstrap starting.
I20260620 11:36:50.021224 19496 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:32799
I20260620 11:36:50.021744 19496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:36:50.022392 19659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:32799 every 8 connection(s)
I20260620 11:36:50.025276 19552 log.cc:824] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Log is configured to *not* fsync() on all Append() calls
I20260620 11:36:50.026424 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 19496
I20260620 11:36:50.026749 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:39745
--local_ip_for_outbound_sockets=127.2.243.130
--tserver_master_addrs=127.2.243.190:40113
--webserver_port=36341
--webserver_interface=127.2.243.130
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:36:50.027640 19552 tablet_bootstrap.cc:492] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:36:50.027915 19552 tablet_bootstrap.cc:492] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Bootstrap complete.
I20260620 11:36:50.029131 19552 ts_tablet_manager.cc:1397] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20260620 11:36:50.030134 19552 raft_consensus.cc:348] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:50.030356 19552 raft_consensus.cc:729] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59c49a3f678c4ebdb8aa4063e5f31a2b, State: Initialized, Role: FOLLOWER
I20260620 11:36:50.030524 19552 consensus_queue.cc:260] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:50.030823 19552 ts_tablet_manager.cc:1428] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260620 11:36:50.101006 19662 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:36:50.022536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.129:45829)
0620 11:36:50.022724 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:50.022766 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:36:50.023580 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:36:50.100802 (+ 77222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":112,"threads_started":1}
W20260620 11:36:50.101280 19660 heartbeater.cc:644] Failed to heartbeat to 127.2.243.190:40113 (0 consecutive failures): Network error: Failed to ping master at 127.2.243.190:40113: Client connection negotiation failed: client connection to 127.2.243.190:40113: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 11:36:50.101622 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:50.101504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.129:48849)
0620 11:36:50.101559 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:50.101601 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 11:36:50.101966 19662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:50.101885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.129:53381)
0620 11:36:50.101931 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:50.101952 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 11:36:50.102072 19660 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 11:36:50.124593 19664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:36:50.124826 19664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:36:50.124866 19664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:36:50.126039 19664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:36:50.126106 19664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:36:50.126154 19664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:36:50.127736 19664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:39745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=36341
--tserver_master_addrs=127.2.243.190:40113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:36:50.128010 19664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:36:50.128234 19664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:36:50.133502 19671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:36:50.133564 19673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:36:50.133632 19664 server_base.cc:1034] running on GCE node
W20260620 11:36:50.133728 19670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:36:50.133960 19664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:36:50.134230 19664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:36:50.135558 19664 hybrid_clock.cc:648] HybridClock initialized: now 1781955410135526 us; error 29 us; skew 500 ppm
Jun 20 11:36:50 dist-test-slave-ctbw krb5kdc[18354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955410, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:36:50.138233 19664 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:36:50.138720 19664 webserver.cc:460] Webserver started at http://127.2.243.130:36341/ using document root <none> and password file <none>
I20260620 11:36:50.139019 19664 fs_manager.cc:359] Metadata directory not provided
I20260620 11:36:50.139092 19664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:36:50.140465 19664 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:36:50.141124 19680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:50.141295 19664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:36:50.141395 19664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "ba680ca73e064a0cbde4f12e7c2c4080"
format_stamp: "Formatted at 2026-06-20 11:36:08 on dist-test-slave-ctbw"
server_key: "e086db3aed0c76b25b9b7115a468500d"
server_key_iv: "cf2d1f3b7efb8c6d5bb68ba680e5b5dc"
server_key_version: "encryptionkey@0"
I20260620 11:36:50.141755 19664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:36:50.154003 19664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:36:50.156435 19664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:36:50.156603 19664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:36:50.156840 19664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:36:50.157297 19687 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 11:36:50.158797 19664 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 11:36:50.158859 19664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 11:36:50.158912 19664 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 11:36:50.159572 19664 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 11:36:50.159631 19664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 11:36:50.159689 19687 tablet_bootstrap.cc:492] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080: Bootstrap starting.
I20260620 11:36:50.161170 19687 log.cc:824] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080: Log is configured to *not* fsync() on all Append() calls
I20260620 11:36:50.164050 19687 tablet_bootstrap.cc:492] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:36:50.164355 19687 tablet_bootstrap.cc:492] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080: Bootstrap complete.
I20260620 11:36:50.164525 19687 ts_tablet_manager.cc:1397] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 11:36:50.165779 19687 raft_consensus.cc:348] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:50.166067 19687 raft_consensus.cc:729] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba680ca73e064a0cbde4f12e7c2c4080, State: Initialized, Role: FOLLOWER
I20260620 11:36:50.166239 19687 consensus_queue.cc:260] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:50.166527 19687 ts_tablet_manager.cc:1428] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260620 11:36:50.166697 19664 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:39745
I20260620 11:36:50.166736 19796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:39745 every 8 connection(s)
I20260620 11:36:50.167097 19664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:36:50.168162 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:50.167735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.130:56815)
0620 11:36:50.167898 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:50.168002 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
W20260620 11:36:50.168341 19797 heartbeater.cc:644] Failed to heartbeat to 127.2.243.190:40113 (0 consecutive failures): Network error: Failed to ping master at 127.2.243.190:40113: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
I20260620 11:36:50.168633 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:50.168548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.130:53063)
0620 11:36:50.168593 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:50.168617 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 11:36:50.168860 19799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:50.168806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.130:58593)
0620 11:36:50.168833 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:50.168849 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260620 11:36:50.168973 19797 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 11:36:50.174187 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 19664
I20260620 11:36:50.174484 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:45031
--local_ip_for_outbound_sockets=127.2.243.131
--tserver_master_addrs=127.2.243.190:40113
--webserver_port=35113
--webserver_interface=127.2.243.131
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:36:50.253644 19800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:36:50.253878 19800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:36:50.253916 19800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:36:50.255049 19800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:36:50.255115 19800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:36:50.255157 19800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:36:50.256567 19800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:45031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=35113
--tserver_master_addrs=127.2.243.190:40113
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:36:50.256830 19800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:36:50.257062 19800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:36:50.261968 19808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:36:50.262060 19800 server_base.cc:1034] running on GCE node
W20260620 11:36:50.261933 19805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:36:50.261933 19806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:36:50.262341 19800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:36:50.262619 19800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:36:50.263761 19800 hybrid_clock.cc:648] HybridClock initialized: now 1781955410263745 us; error 34 us; skew 500 ppm
Jun 20 11:36:50 dist-test-slave-ctbw krb5kdc[18354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955410, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:36:50.266149 19800 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:36:50.266600 19800 webserver.cc:460] Webserver started at http://127.2.243.131:35113/ using document root <none> and password file <none>
I20260620 11:36:50.266826 19800 fs_manager.cc:359] Metadata directory not provided
I20260620 11:36:50.266923 19800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:36:50.268215 19800 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:36:50.268774 19815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:50.268934 19800 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:36:50.269035 19800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "d7b3217648914448b7eaa923a4e7d3d3"
format_stamp: "Formatted at 2026-06-20 11:36:08 on dist-test-slave-ctbw"
server_key: "2731c599644d514da4476b31863bb525"
server_key_iv: "4bc261953bb41bcf6ded55b304ee65d4"
server_key_version: "encryptionkey@0"
I20260620 11:36:50.269359 19800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:36:50.282351 19800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:36:50.284148 19800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:36:50.284308 19800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:36:50.284560 19800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:36:50.284899 19800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:36:50.284958 19800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:50.284991 19800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:36:50.285020 19800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:36:50.291023 19800 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:45031
I20260620 11:36:50.291062 19928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:45031 every 8 connection(s)
I20260620 11:36:50.291407 19800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:36:50.292505 19931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:50.292144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.131:51363)
0620 11:36:50.292293 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:50.292366 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
W20260620 11:36:50.292919 19929 heartbeater.cc:644] Failed to heartbeat to 127.2.243.190:40113 (0 consecutive failures): Network error: Failed to ping master at 127.2.243.190:40113: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
I20260620 11:36:50.293196 19931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:50.293111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.131:47599)
0620 11:36:50.293153 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:50.293179 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 11:36:50.293524 19931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:50.293430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.131:53499)
0620 11:36:50.293470 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:50.293510 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 11:36:50.293643 19929 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 11:36:50.299402 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 19800
I20260620 11:36:51.102643 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:51.102423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.129:52111)
0620 11:36:51.102544 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:51.102579 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260620 11:36:51.169463 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:51.169261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.130:51035)
0620 11:36:51.169386 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:51.169421 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":55,"threads_started":1}
I20260620 11:36:51.294181 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:51.293956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.131:60571)
0620 11:36:51.294087 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:51.294125 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":73,"threads_started":1}
I20260620 11:36:51.631102 19936 raft_consensus.cc:482] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:36:51.631244 19936 raft_consensus.cc:504] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:51.631532 19936 leader_election.cc:288] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d7b3217648914448b7eaa923a4e7d3d3 (127.2.243.131:45031), 59c49a3f678c4ebdb8aa4063e5f31a2b (127.2.243.129:32799)
Jun 20 11:36:51 dist-test-slave-ctbw krb5kdc[18354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955410, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.131@KRBTEST.COM
Jun 20 11:36:51 dist-test-slave-ctbw krb5kdc[18354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955410, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.129@KRBTEST.COM
I20260620 11:36:51.642266 19938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:51.631869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:34661 (local address 127.2.243.129:32799)
0620 11:36:51.632100 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:51.632105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:51.632117 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:36:51.632230 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:51.632232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:51.632261 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:51.632352 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:36:51.633026 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:51.633971 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:51.635118 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:51.635534 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:51.635658 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:51.638387 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:36:51.638390 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:36:51.638391 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:36:51.638400 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:36:51.640277 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:51.640827 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:51.640830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:51.640830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:51.640951 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:51.641279 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:51.641280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:51.641280 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:51.641593 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:36:51.641680 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:51.642033 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 11:36:51.642170 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":71,"threads_started":1}
I20260620 11:36:51.642277 19939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:51.631881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:53315 (local address 127.2.243.131:45031)
0620 11:36:51.632092 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:51.632111 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:51.632143 (+ 32us) server_negotiation.cc:407] Connection header received
0620 11:36:51.632589 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:51.632591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:51.632623 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:51.632771 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:36:51.633423 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:51.634247 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:51.635185 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:51.635571 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:51.635695 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:51.637976 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:36:51.637979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:36:51.637979 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:36:51.637989 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:36:51.640092 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:51.640827 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:51.640830 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:51.640830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:51.640951 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:51.641282 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:51.641283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:51.641283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:51.641593 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:36:51.641680 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:51.642033 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 11:36:51.642162 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":66,"threads_started":1}
I20260620 11:36:51.642349 19937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:51.631714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:45031 (local address 127.2.243.130:53315)
0620 11:36:51.631850 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:51.631861 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:36:51.632014 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:36:51.632745 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:36:51.632746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:36:51.633240 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:36:51.633299 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:51.633301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:51.634344 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:51.634345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:51.635101 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:51.635103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:51.635673 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:51.635674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:51.635767 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:51.636345 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:36:51.636353 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:36:51.637874 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:36:51.640249 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:36:51.640254 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:36:51.640256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:36:51.640737 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:36:51.641028 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:36:51.641029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:36:51.641029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:36:51.641190 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:36:51.641681 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:36:51.641681 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:36:51.641847 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 11:36:51.641963 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 11:36:51.642167 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":53,"thread_start_us":46,"threads_started":1}
I20260620 11:36:51.642428 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:51.631714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:32799 (local address 127.2.243.130:34661)
0620 11:36:51.631776 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:51.631816 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 11:36:51.632019 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:36:51.632407 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:36:51.632411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:36:51.632701 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:36:51.632900 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:51.632906 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:51.634075 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:51.634076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:51.635033 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:51.635036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:51.635627 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:51.635629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:51.635766 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:51.636436 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:36:51.636441 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:36:51.638266 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:36:51.640357 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:36:51.640358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:36:51.640358 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:36:51.640737 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:36:51.641028 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:36:51.641029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:36:51.641029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:36:51.641190 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:36:51.641674 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:36:51.641680 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:36:51.641847 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 11:36:51.641963 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 11:36:51.642221 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":150}
I20260620 11:36:51.642901 19614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b166b43f77a24dd3900977a8833fe962" candidate_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" is_pre_election: true
I20260620 11:36:51.643049 19614 raft_consensus.cc:2454] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba680ca73e064a0cbde4f12e7c2c4080 in term 1.
I20260620 11:36:51.643406 19683 leader_election.cc:302] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59c49a3f678c4ebdb8aa4063e5f31a2b, ba680ca73e064a0cbde4f12e7c2c4080; no voters:
I20260620 11:36:51.643566 19936 raft_consensus.cc:2781] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 11:36:51.643689 19936 raft_consensus.cc:482] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:36:51.643719 19936 raft_consensus.cc:3037] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [term 1 FOLLOWER]: Advancing to term 2
W20260620 11:36:51.643888 19684 leader_election.cc:341] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d7b3217648914448b7eaa923a4e7d3d3 (127.2.243.131:45031): Not found: Tablet not found: b166b43f77a24dd3900977a8833fe962
I20260620 11:36:51.645074 19936 raft_consensus.cc:504] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:51.645251 19936 leader_election.cc:288] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [CANDIDATE]: Term 2 election: Requested vote from peers d7b3217648914448b7eaa923a4e7d3d3 (127.2.243.131:45031), 59c49a3f678c4ebdb8aa4063e5f31a2b (127.2.243.129:32799)
I20260620 11:36:51.645390 19614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b166b43f77a24dd3900977a8833fe962" candidate_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b"
I20260620 11:36:51.645473 19614 raft_consensus.cc:3037] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 1 FOLLOWER]: Advancing to term 2
W20260620 11:36:51.645583 19684 leader_election.cc:341] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer d7b3217648914448b7eaa923a4e7d3d3 (127.2.243.131:45031): Not found: Tablet not found: b166b43f77a24dd3900977a8833fe962
I20260620 11:36:51.646531 19614 raft_consensus.cc:2454] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba680ca73e064a0cbde4f12e7c2c4080 in term 2.
I20260620 11:36:51.646790 19683 leader_election.cc:302] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59c49a3f678c4ebdb8aa4063e5f31a2b, ba680ca73e064a0cbde4f12e7c2c4080; no voters: d7b3217648914448b7eaa923a4e7d3d3
I20260620 11:36:51.646934 19936 raft_consensus.cc:2781] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 11:36:51.647058 19936 raft_consensus.cc:686] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [term 2 LEADER]: Becoming Leader. State: Replica: ba680ca73e064a0cbde4f12e7c2c4080, State: Running, Role: LEADER
I20260620 11:36:51.647148 19936 consensus_queue.cc:237] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:51.647397 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:51.647290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.130:51005)
0620 11:36:51.647332 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:51.647357 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 11:36:52.103353 19942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:52.103096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.129:54837)
0620 11:36:52.103251 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:52.103307 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":79,"threads_started":1}
W20260620 11:36:52.174541 19684 consensus_peers.cc:487] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 -> Peer d7b3217648914448b7eaa923a4e7d3d3 (127.2.243.131:45031): Couldn't send request to peer d7b3217648914448b7eaa923a4e7d3d3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b166b43f77a24dd3900977a8833fe962. This is attempt 1: this message will repeat every 5th retry.
I20260620 11:36:52.175246 19614 raft_consensus.cc:1261] T b166b43f77a24dd3900977a8833fe962 P 59c49a3f678c4ebdb8aa4063e5f31a2b [term 2 FOLLOWER]: Refusing update from remote peer ba680ca73e064a0cbde4f12e7c2c4080: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 11:36:52.175603 19943 consensus_queue.cc:1035] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 11:36:52.176460 19942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:52.176346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.129:52395)
0620 11:36:52.176398 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:52.176432 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 11:36:52.177388 19949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:52.177210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.130:33735)
0620 11:36:52.177324 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:52.177351 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":79,"threads_started":1}
I20260620 11:36:52.177392 19942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:52.177260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.129:33673)
0620 11:36:52.177319 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:52.177357 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 11:36:52.294821 19950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:52.294624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.131:55019)
0620 11:36:52.294755 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:52.294784 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260620 11:36:52.745391 19952 ts_tablet_manager.cc:927] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3: Initiating tablet copy from peer ba680ca73e064a0cbde4f12e7c2c4080 (127.2.243.130:39745)
I20260620 11:36:52.745597 19952 tablet_copy_client.cc:316] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3: tablet copy: Beginning tablet copy session from remote peer at address 127.2.243.130:39745
Jun 20 11:36:52 dist-test-slave-ctbw krb5kdc[18354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955410, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for kudu/127.2.243.130@KRBTEST.COM
I20260620 11:36:52.753340 19954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:52.745914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:36495 (local address 127.2.243.130:39745)
0620 11:36:52.746049 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 11:36:52.746053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:36:52.746060 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:36:52.746082 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:36:52.746083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:36:52.746095 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:36:52.746147 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:36:52.746563 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:52.747357 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:52.748088 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:36:52.748382 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:36:52.748434 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:52.750347 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:36:52.750353 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:36:52.750353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:36:52.750360 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:36:52.752117 (+ 1757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:52.752598 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:52.752601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:52.752602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:52.752661 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:36:52.752858 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:36:52.752859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:36:52.752859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:36:52.752988 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:36:52.753055 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 11:36:52.753212 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 11:36:52.753273 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":50,"threads_started":1}
I20260620 11:36:52.753410 19950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:52.745823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:39745 (local address 127.2.243.131:36495)
0620 11:36:52.745871 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:52.745889 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 11:36:52.745949 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:36:52.746173 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:36:52.746178 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:36:52.746428 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:36:52.746489 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:52.746491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:52.747434 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:52.747435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:52.748017 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:36:52.748019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:36:52.748444 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:36:52.748445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:36:52.748499 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:36:52.748953 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:36:52.748957 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:36:52.750254 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:36:52.752246 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:36:52.752255 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:36:52.752258 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:36:52.752513 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:36:52.752732 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:36:52.752732 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:36:52.752733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:36:52.752800 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:36:52.753050 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:36:52.753056 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:36:52.753107 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 11:36:52.753190 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 11:36:52.753292 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 11:36:52.753872 19770 tablet_copy_service.cc:140] P ba680ca73e064a0cbde4f12e7c2c4080: Received BeginTabletCopySession request for tablet b166b43f77a24dd3900977a8833fe962 from peer d7b3217648914448b7eaa923a4e7d3d3 ({username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:36495)
I20260620 11:36:52.753934 19770 tablet_copy_service.cc:161] P ba680ca73e064a0cbde4f12e7c2c4080: Beginning new tablet copy session on tablet b166b43f77a24dd3900977a8833fe962 from peer d7b3217648914448b7eaa923a4e7d3d3 at {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:36495: session id = d7b3217648914448b7eaa923a4e7d3d3-b166b43f77a24dd3900977a8833fe962
I20260620 11:36:52.754398 19770 tablet_copy_source_session.cc:205] T b166b43f77a24dd3900977a8833fe962 P ba680ca73e064a0cbde4f12e7c2c4080: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 11:36:52.755388 19952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b166b43f77a24dd3900977a8833fe962. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:36:52.757910 19952 tablet_copy_client.cc:791] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3: tablet copy: Starting download of 0 data blocks...
I20260620 11:36:52.758119 19952 tablet_copy_client.cc:655] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3: tablet copy: Starting download of 1 WAL segments...
I20260620 11:36:52.758982 19952 tablet_copy_client.cc:531] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 11:36:52.759651 19952 tablet_bootstrap.cc:492] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3: Bootstrap starting.
I20260620 11:36:52.760941 19952 log.cc:824] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3: Log is configured to *not* fsync() on all Append() calls
I20260620 11:36:52.763494 19952 tablet_bootstrap.cc:492] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:36:52.763718 19952 tablet_bootstrap.cc:492] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3: Bootstrap complete.
I20260620 11:36:52.763875 19952 ts_tablet_manager.cc:1397] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260620 11:36:52.764755 19952 raft_consensus.cc:348] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:52.764974 19952 raft_consensus.cc:729] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7b3217648914448b7eaa923a4e7d3d3, State: Initialized, Role: FOLLOWER
I20260620 11:36:52.765133 19952 consensus_queue.cc:260] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7b3217648914448b7eaa923a4e7d3d3" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45031 } } peers { permanent_uuid: "ba680ca73e064a0cbde4f12e7c2c4080" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39745 } } peers { permanent_uuid: "59c49a3f678c4ebdb8aa4063e5f31a2b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 32799 } }
I20260620 11:36:52.765413 19952 ts_tablet_manager.cc:1428] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:36:52.765622 19950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:52.765543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.131:57403)
0620 11:36:52.765574 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:52.765598 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260620 11:36:52.765632 19770 tablet_copy_service.cc:342] P ba680ca73e064a0cbde4f12e7c2c4080: Request end of tablet copy session d7b3217648914448b7eaa923a4e7d3d3-b166b43f77a24dd3900977a8833fe962 received from {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:36495
I20260620 11:36:52.765686 19770 tablet_copy_service.cc:433] P ba680ca73e064a0cbde4f12e7c2c4080: ending tablet copy session d7b3217648914448b7eaa923a4e7d3d3-b166b43f77a24dd3900977a8833fe962 on tablet b166b43f77a24dd3900977a8833fe962 with peer d7b3217648914448b7eaa923a4e7d3d3
I20260620 11:36:52.765950 19950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:52.765852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.131:48583)
0620 11:36:52.765899 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:52.765922 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 11:36:53.177989 19957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:53.177769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.130:45493)
0620 11:36:53.177885 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:53.177948 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":49,"threads_started":1}
I20260620 11:36:53.177990 19958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:53.177767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.129:38029)
0620 11:36:53.177912 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:53.177952 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260620 11:36:53.358397 19957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:53.358272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.130:58957)
0620 11:36:53.358315 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:53.358342 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 11:36:53.358417 19883 raft_consensus.cc:1203] T b166b43f77a24dd3900977a8833fe962 P d7b3217648914448b7eaa923a4e7d3d3 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260620 11:36:53.359062 19963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:53.358907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.131:48165)
0620 11:36:53.358985 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:53.359020 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62,"thread_start_us":42,"threads_started":1}
I20260620 11:36:54.178659 19964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:54.178459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.129:57887)
0620 11:36:54.178577 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:54.178615 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260620 11:36:54.359017 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:54.358802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.130:53997)
0620 11:36:54.358942 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:54.358973 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":100,"threads_started":1}
I20260620 11:36:54.359653 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:54.359439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.131:44739)
0620 11:36:54.359581 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:54.359617 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":49,"threads_started":1}
I20260620 11:36:55.179392 19967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:55.179196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.129:39219)
0620 11:36:55.179308 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:55.179341 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":59,"threads_started":1}
I20260620 11:36:55.359730 19969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:55.359488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.130:34043)
0620 11:36:55.359645 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:55.359684 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":73,"threads_started":1}
I20260620 11:36:55.360338 19970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:55.360153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.131:39541)
0620 11:36:55.360265 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:55.360299 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260620 11:36:56.179957 19971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:56.179771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.129:48793)
0620 11:36:56.179882 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:56.179916 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
I20260620 11:36:56.360445 19972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:56.360217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.130:33631)
0620 11:36:56.360358 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:56.360397 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20260620 11:36:56.361011 19973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:56.360793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.131:42243)
0620 11:36:56.360928 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:56.360970 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":95,"threads_started":1}
I20260620 11:36:57.180629 19974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:57.180399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.129:60181)
0620 11:36:57.180537 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:57.180576 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260620 11:36:57.361088 19975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:57.360883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.130:53679)
0620 11:36:57.361008 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:57.361042 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260620 11:36:57.361615 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:57.361418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.131:46605)
0620 11:36:57.361542 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:57.361579 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":51,"threads_started":1}
I20260620 11:36:58.181249 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:58.181049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.129:55533)
0620 11:36:58.181163 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:58.181198 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
I20260620 11:36:58.362010 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:58.361733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.130:49453)
0620 11:36:58.361882 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:58.361943 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":63,"threads_started":1}
I20260620 11:36:58.362341 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:58.362036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.131:45689)
0620 11:36:58.362177 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:58.362248 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
I20260620 11:36:59.181973 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:59.181780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.129:56663)
0620 11:36:59.181899 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:59.181930 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260620 11:36:59.362700 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:59.362465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.130:54533)
0620 11:36:59.362619 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:59.362656 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
I20260620 11:36:59.363117 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:36:59.362852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.131:45833)
0620 11:36:59.363024 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 11:36:59.363065 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
I20260620 11:37:00.182600 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:00.182386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.129:44325)
0620 11:37:00.182508 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:00.182556 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":57,"threads_started":1}
I20260620 11:37:00.363505 19984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:00.363213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.130:37731)
0620 11:37:00.363391 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:00.363439 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":71,"threads_started":1}
I20260620 11:37:00.363736 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:00.363530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.131:46225)
0620 11:37:00.363674 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:00.363701 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":66,"threads_started":1}
I20260620 11:37:01.183300 19987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:01.183046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.129:50093)
0620 11:37:01.183179 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:01.183225 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
I20260620 11:37:01.364374 19988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:01.364080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.130:47629)
0620 11:37:01.364262 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:01.364316 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":83,"threads_started":1}
I20260620 11:37:01.364676 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:01.364345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.131:44697)
0620 11:37:01.364496 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:01.364544 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":63,"threads_started":1}
I20260620 11:37:02.183943 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:02.183739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.129:43813)
0620 11:37:02.183861 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:02.183898 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260620 11:37:02.365197 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:02.364912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.130:48565)
0620 11:37:02.365052 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:02.365129 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
I20260620 11:37:02.365406 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:02.365171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.131:49039)
0620 11:37:02.365296 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:02.365348 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260620 11:37:03.184765 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:03.184536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.129:45371)
0620 11:37:03.184680 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:03.184719 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":75,"threads_started":1}
I20260620 11:37:03.365947 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:03.365717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.130:43365)
0620 11:37:03.365848 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:03.365888 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":58,"threads_started":1}
I20260620 11:37:03.366024 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:03.365791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.131:47099)
0620 11:37:03.365948 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:03.365984 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":80,"threads_started":1}
I20260620 11:37:04.185467 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:04.185269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.129:33517)
0620 11:37:04.185387 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:04.185421 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
I20260620 11:37:04.366691 19997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:04.366457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.130:41385)
0620 11:37:04.366587 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:04.366628 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":57,"threads_started":1}
I20260620 11:37:04.366737 19998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:04.366503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.131:41943)
0620 11:37:04.366662 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:04.366697 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":101,"threads_started":1}
I20260620 11:37:05.186241 19999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:05.186001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:40113 (local address 127.2.243.129:46603)
0620 11:37:05.186149 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:05.186196 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.190:40113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":65,"threads_started":1}
I20260620 11:37:05.301779 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 19496
I20260620 11:37:05.306447 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 19664
I20260620 11:37:05.310961 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 19800
2026-06-20T11:37:05Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58202 ms)
[----------] 30 tests from SecurityITest (232492 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:05 dist-test-slave-ctbw krb5kdc[20002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:05 dist-test-slave-ctbw krb5kdc[20002](info): set up 2 sockets
Jun 20 11:37:05 dist-test-slave-ctbw krb5kdc[20002](info): commencing operation
krb5kdc: starting...
W20260620 11:37:06.350773 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:06 dist-test-slave-ctbw krb5kdc[20002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:37:06.463210 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:33047
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:36353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:33047
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:06.543995 20015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:06.544284 20015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:06.544332 20015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:06.545501 20015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:06.545569 20015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:06.545593 20015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:06.545620 20015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:06.545650 20015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:06.547639 20015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:36353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:33047
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:33047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:06.548041 20015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:06.548303 20015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:06.553704 20020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:06.553714 20021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:06.553740 20015 server_base.cc:1034] running on GCE node
W20260620 11:37:06.553721 20023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:06.554141 20015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:06.554468 20015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:06.555608 20015 hybrid_clock.cc:648] HybridClock initialized: now 1781955426555582 us; error 43 us; skew 500 ppm
Jun 20 11:37:06 dist-test-slave-ctbw krb5kdc[20002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955426, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:06.562355 20015 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:37:06.562846 20015 webserver.cc:460] Webserver started at http://127.2.243.190:36217/ using document root <none> and password file <none>
I20260620 11:37:06.563176 20015 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:06.563225 20015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:06.563388 20015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:06.564488 20015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "240ac80952724e9f97ec1f8715b98af4"
format_stamp: "Formatted at 2026-06-20 11:37:06 on dist-test-slave-ctbw"
server_key: "06ae9172ece90f83ed29762c3f74e77b"
server_key_iv: "921800d55a60b4bfcb6e634de358b860"
server_key_version: "encryptionkey@0"
I20260620 11:37:06.564857 20015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "240ac80952724e9f97ec1f8715b98af4"
format_stamp: "Formatted at 2026-06-20 11:37:06 on dist-test-slave-ctbw"
server_key: "06ae9172ece90f83ed29762c3f74e77b"
server_key_iv: "921800d55a60b4bfcb6e634de358b860"
server_key_version: "encryptionkey@0"
I20260620 11:37:06.566216 20015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:37:06.567068 20030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:06.567277 20015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:37:06.567371 20015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "240ac80952724e9f97ec1f8715b98af4"
format_stamp: "Formatted at 2026-06-20 11:37:06 on dist-test-slave-ctbw"
server_key: "06ae9172ece90f83ed29762c3f74e77b"
server_key_iv: "921800d55a60b4bfcb6e634de358b860"
server_key_version: "encryptionkey@0"
I20260620 11:37:06.567444 20015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:06.583514 20015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:06.585795 20015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:06.585969 20015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:06.590214 20015 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:33047
I20260620 11:37:06.590236 20082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:33047 every 8 connection(s)
I20260620 11:37:06.590610 20015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:06.591598 20083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:06.593791 20083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4: Bootstrap starting.
I20260620 11:37:06.594375 20083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:06.594657 20083 log.cc:824] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:06.595436 20083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4: No bootstrap required, opened a new log
I20260620 11:37:06.596980 20083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240ac80952724e9f97ec1f8715b98af4" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33047 } }
I20260620 11:37:06.597113 20083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:06.597163 20083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 240ac80952724e9f97ec1f8715b98af4, State: Initialized, Role: FOLLOWER
I20260620 11:37:06.597311 20083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240ac80952724e9f97ec1f8715b98af4" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33047 } }
I20260620 11:37:06.597373 20083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:06.597430 20083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:06.597479 20083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:06.597616 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 20015
I20260620 11:37:06.597735 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:06.597942 3022 external_mini_cluster.cc:1428] Setting key 2c84bb58c6c325a9c7035c06155ecd51
I20260620 11:37:06.598133 20083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240ac80952724e9f97ec1f8715b98af4" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33047 } }
I20260620 11:37:06.598245 20083 leader_election.cc:302] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 240ac80952724e9f97ec1f8715b98af4; no voters:
I20260620 11:37:06.598459 20083 leader_election.cc:288] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:06.598618 20089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:06.598714 20083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:06.598747 20089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4 [term 1 LEADER]: Becoming Leader. State: Replica: 240ac80952724e9f97ec1f8715b98af4, State: Running, Role: LEADER
I20260620 11:37:06.598960 20089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240ac80952724e9f97ec1f8715b98af4" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33047 } }
I20260620 11:37:06.599526 20087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "240ac80952724e9f97ec1f8715b98af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240ac80952724e9f97ec1f8715b98af4" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33047 } } }
I20260620 11:37:06.599618 20087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:06.599917 20097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:06.600047 20090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 240ac80952724e9f97ec1f8715b98af4. Latest consensus state: current_term: 1 leader_uuid: "240ac80952724e9f97ec1f8715b98af4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "240ac80952724e9f97ec1f8715b98af4" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33047 } } }
I20260620 11:37:06.600193 20090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:06.600744 20097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:06.602974 20097 catalog_manager.cc:1347] Generated new cluster ID: dab3e272d9d446f9bef040b1741424da
I20260620 11:37:06.603071 20097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:37:06 dist-test-slave-ctbw krb5kdc[20002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:37:06.608251 20097 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:06.608798 20097 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:06.610211 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:06.598386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53900 (local address 127.2.243.190:33047)
0620 11:37:06.599024 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:06.599033 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:06.599067 (+ 34us) server_negotiation.cc:407] Connection header received
0620 11:37:06.599177 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:06.599182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:06.599223 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:06.600209 (+ 986us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:06.600626 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:06.601939 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:06.603036 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:06.603594 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:06.603694 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:06.606488 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:06.606492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:06.606493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:06.606504 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:06.608612 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:06.609022 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:06.609025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:06.609025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:06.609128 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:06.609337 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:06.609338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:06.609338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:06.609625 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:06.609791 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:06.609863 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 11:37:06.610008 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":75,"threads_started":1}
I20260620 11:37:06.613150 20097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4: Generated new TSK 0
I20260620 11:37:06.613318 20097 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
I20260620 11:37:06.641074 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33047
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:36353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:06.735399 20109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:06.735656 20109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:06.735692 20109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:06.736828 20109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:06.736897 20109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:06.736943 20109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:37:06.738361 20109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:36353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:06.738611 20109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:06.738843 20109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:06.743829 20114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:06.743840 20115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:06.743960 20117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:06.744114 20109 server_base.cc:1034] running on GCE node
I20260620 11:37:06.744289 20109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:06.744524 20109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:06.745680 20109 hybrid_clock.cc:648] HybridClock initialized: now 1781955426745666 us; error 31 us; skew 500 ppm
Jun 20 11:37:06 dist-test-slave-ctbw krb5kdc[20002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955426, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:06.748303 20109 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:37:06.748728 20109 webserver.cc:460] Webserver started at http://127.2.243.129:45757/ using document root <none> and password file <none>
I20260620 11:37:06.748942 20109 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:06.748987 20109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:06.749122 20109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:06.750139 20109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "26103ea752f24a27aea1ae095b1d8425"
format_stamp: "Formatted at 2026-06-20 11:37:06 on dist-test-slave-ctbw"
server_key: "62fc0f8664012654b2a3dfee35f36ee4"
server_key_iv: "367513393ded7ce7187429d168266d86"
server_key_version: "encryptionkey@0"
I20260620 11:37:06.750507 20109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "26103ea752f24a27aea1ae095b1d8425"
format_stamp: "Formatted at 2026-06-20 11:37:06 on dist-test-slave-ctbw"
server_key: "62fc0f8664012654b2a3dfee35f36ee4"
server_key_iv: "367513393ded7ce7187429d168266d86"
server_key_version: "encryptionkey@0"
I20260620 11:37:06.751904 20109 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:37:06.752689 20124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:06.752908 20109 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:37:06.753065 20109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "26103ea752f24a27aea1ae095b1d8425"
format_stamp: "Formatted at 2026-06-20 11:37:06 on dist-test-slave-ctbw"
server_key: "62fc0f8664012654b2a3dfee35f36ee4"
server_key_iv: "367513393ded7ce7187429d168266d86"
server_key_version: "encryptionkey@0"
I20260620 11:37:06.753141 20109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:06.767421 20109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:06.769174 20109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:06.769336 20109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:06.769593 20109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:37:06.769933 20109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:37:06.769975 20109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:06.770010 20109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:37:06.770037 20109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:06.776048 20109 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:38709
I20260620 11:37:06.776085 20237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:38709 every 8 connection(s)
I20260620 11:37:06.776415 20109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
Jun 20 11:37:06 dist-test-slave-ctbw krb5kdc[20002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955426, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:37:06.785836 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:06.777260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:41427 (local address 127.2.243.190:33047)
0620 11:37:06.777353 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:06.777356 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:06.777545 (+ 189us) server_negotiation.cc:407] Connection header received
0620 11:37:06.778087 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:06.778088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:06.778100 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:06.778151 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:06.779194 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:06.779709 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:06.780720 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:06.780930 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:06.781007 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:06.783223 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:06.783225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:06.783225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:06.783231 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:06.784654 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:06.785016 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:06.785016 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:06.785017 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:06.785065 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:06.785290 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:06.785291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:06.785291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:06.785389 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:06.785438 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:06.785706 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 11:37:06.785772 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 11:37:06.786028 20240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:06.777256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33047 (local address 127.2.243.129:41427)
0620 11:37:06.777415 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:06.777461 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:06.777979 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:06.778232 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:06.778236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:06.778808 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:06.779071 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:06.779076 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:06.779814 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:06.779815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:06.780610 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:06.780612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:06.781013 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:06.781014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:06.781138 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:06.781588 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:06.781596 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:06.783133 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:06.784726 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:06.784731 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:06.784732 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:06.784951 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:06.785122 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:06.785122 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:06.785123 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:06.785216 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:06.785443 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:06.785447 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:06.785583 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 11:37:06.785676 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 11:37:06.785796 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260620 11:37:06.786227 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 20109
I20260620 11:37:06.786327 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:37:06.786518 3022 external_mini_cluster.cc:1428] Setting key 48d625ac4e2b0c7e9889f5c41fd944ce
I20260620 11:37:06.786537 20238 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33047
I20260620 11:37:06.786633 20238 heartbeater.cc:464] Registering TS with master...
I20260620 11:37:06.786827 20238 heartbeater.cc:511] Master 127.2.243.190:33047 requested a full tablet report, sending...
I20260620 11:37:06.787377 20047 ts_manager.cc:194] Registered new tserver with Master: 26103ea752f24a27aea1ae095b1d8425 (127.2.243.129:38709)
I20260620 11:37:06.787976 20047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:41427
WARNING: no policy specified for kudu/127.2.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
I20260620 11:37:06.809540 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33047
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:36353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:06.888106 20243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:06.888332 20243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:06.888368 20243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:06.889591 20243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:06.889654 20243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:06.889698 20243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:37:06.891165 20243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:36353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:06.891429 20243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:06.891673 20243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:06.896415 20251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:06.896414 20249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:06.896576 20243 server_base.cc:1034] running on GCE node
W20260620 11:37:06.896421 20248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:06.896881 20243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:06.897107 20243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:06.898257 20243 hybrid_clock.cc:648] HybridClock initialized: now 1781955426898243 us; error 48 us; skew 500 ppm
Jun 20 11:37:06 dist-test-slave-ctbw krb5kdc[20002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955426, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:06.900835 20243 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:37:06.901178 20243 webserver.cc:460] Webserver started at http://127.2.243.130:38827/ using document root <none> and password file <none>
I20260620 11:37:06.901346 20243 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:06.901384 20243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:06.901485 20243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:06.902446 20243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "dcb274c89c31479f89e4b186f70648d6"
format_stamp: "Formatted at 2026-06-20 11:37:06 on dist-test-slave-ctbw"
server_key: "0fa60cbc8d4d1d1ad0e5178be0e37ee0"
server_key_iv: "638600d7a0d7bcb7f609fa19ee34a279"
server_key_version: "encryptionkey@0"
I20260620 11:37:06.902727 20243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "dcb274c89c31479f89e4b186f70648d6"
format_stamp: "Formatted at 2026-06-20 11:37:06 on dist-test-slave-ctbw"
server_key: "0fa60cbc8d4d1d1ad0e5178be0e37ee0"
server_key_iv: "638600d7a0d7bcb7f609fa19ee34a279"
server_key_version: "encryptionkey@0"
I20260620 11:37:06.904040 20243 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:37:06.904661 20258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:06.904873 20243 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:37:06.904983 20243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "dcb274c89c31479f89e4b186f70648d6"
format_stamp: "Formatted at 2026-06-20 11:37:06 on dist-test-slave-ctbw"
server_key: "0fa60cbc8d4d1d1ad0e5178be0e37ee0"
server_key_iv: "638600d7a0d7bcb7f609fa19ee34a279"
server_key_version: "encryptionkey@0"
I20260620 11:37:06.905056 20243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:06.914857 20243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:06.916662 20243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:06.916822 20243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:06.917029 20243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:37:06.917317 20243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:37:06.917351 20243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:06.917371 20243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:37:06.917416 20243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:06.922957 20243 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:39409
I20260620 11:37:06.923008 20371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:39409 every 8 connection(s)
I20260620 11:37:06.923300 20243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:37:06.923626 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 20243
I20260620 11:37:06.923748 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:37:06.923983 3022 external_mini_cluster.cc:1428] Setting key 258c2696a7673730facf3da1cac954ca
Jun 20 11:37:06 dist-test-slave-ctbw krb5kdc[20002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955426, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.243.131@KRBTEST.COM; defaulting to no policy
I20260620 11:37:06.933118 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:06.924088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:53567 (local address 127.2.243.190:33047)
0620 11:37:06.924150 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:06.924153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:06.924438 (+ 285us) server_negotiation.cc:407] Connection header received
0620 11:37:06.925026 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:06.925027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:06.925039 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:06.925089 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:06.926155 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:06.926634 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:06.927713 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:06.928055 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:06.928138 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:06.930188 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:06.930190 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:06.930190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:06.930199 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:06.931821 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:06.932316 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:06.932317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:06.932318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:06.932379 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:06.932599 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:06.932600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:06.932600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:06.932701 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:06.932755 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:06.932963 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 11:37:06.933055 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 11:37:06.933400 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:06.924063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33047 (local address 127.2.243.130:53567)
0620 11:37:06.924315 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:06.924358 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:06.924908 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:06.925167 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:06.925175 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:06.925736 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:06.926054 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:06.926059 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:06.926724 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:06.926725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:06.927623 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:06.927625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:06.928138 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:06.928139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:06.928235 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:06.928757 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:06.928766 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:06.930101 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:06.931915 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:06.931922 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:06.931924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:06.932227 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:06.932454 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:06.932454 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:06.932454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:06.932545 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:06.932757 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:06.932760 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:06.932861 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 11:37:06.932938 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 11:37:06.933119 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":81,"threads_started":1}
I20260620 11:37:06.933960 20372 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33047
I20260620 11:37:06.934041 20372 heartbeater.cc:464] Registering TS with master...
I20260620 11:37:06.934170 20372 heartbeater.cc:511] Master 127.2.243.190:33047 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.131@KRBTEST.COM" created.
I20260620 11:37:06.934538 20047 ts_manager.cc:194] Registered new tserver with Master: dcb274c89c31479f89e4b186f70648d6 (127.2.243.130:39409)
I20260620 11:37:06.934928 20047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:53567
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
I20260620 11:37:06.947755 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33047
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:36353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:07.028682 20377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:07.028873 20377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:07.028895 20377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:07.029947 20377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:07.029982 20377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:07.030009 20377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:37:07.031418 20377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:36353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33047
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:07.031603 20377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:07.031769 20377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:07.036459 20382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:07.036484 20383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:07.036459 20385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:07.036720 20377 server_base.cc:1034] running on GCE node
I20260620 11:37:07.036918 20377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:07.037142 20377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:07.038314 20377 hybrid_clock.cc:648] HybridClock initialized: now 1781955427038294 us; error 46 us; skew 500 ppm
Jun 20 11:37:07 dist-test-slave-ctbw krb5kdc[20002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955427, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:07.040711 20377 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:37:07.041167 20377 webserver.cc:460] Webserver started at http://127.2.243.131:36037/ using document root <none> and password file <none>
I20260620 11:37:07.041404 20377 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:07.041472 20377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:07.041608 20377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:07.042555 20377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "3c9a05e587804c8fa3c091f84ef9ab7c"
format_stamp: "Formatted at 2026-06-20 11:37:07 on dist-test-slave-ctbw"
server_key: "74aafe999e834df01240a391b4bb7c3c"
server_key_iv: "fd25e9c040c675a51b8d2647ccac7590"
server_key_version: "encryptionkey@0"
I20260620 11:37:07.042907 20377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "3c9a05e587804c8fa3c091f84ef9ab7c"
format_stamp: "Formatted at 2026-06-20 11:37:07 on dist-test-slave-ctbw"
server_key: "74aafe999e834df01240a391b4bb7c3c"
server_key_iv: "fd25e9c040c675a51b8d2647ccac7590"
server_key_version: "encryptionkey@0"
I20260620 11:37:07.044183 20377 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:37:07.044914 20392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:07.045112 20377 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:37:07.045186 20377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "3c9a05e587804c8fa3c091f84ef9ab7c"
format_stamp: "Formatted at 2026-06-20 11:37:07 on dist-test-slave-ctbw"
server_key: "74aafe999e834df01240a391b4bb7c3c"
server_key_iv: "fd25e9c040c675a51b8d2647ccac7590"
server_key_version: "encryptionkey@0"
I20260620 11:37:07.045243 20377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:07.055214 20377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:07.057564 20377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:07.057672 20377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:07.057842 20377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:37:07.058100 20377 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:37:07.058130 20377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:07.058151 20377 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:37:07.058163 20377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:07.063859 20377 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:34769
I20260620 11:37:07.063933 20505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:34769 every 8 connection(s)
I20260620 11:37:07.064224 20377 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
Jun 20 11:37:07 dist-test-slave-ctbw krb5kdc[20002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955427, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:37:07.072057 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 20377
I20260620 11:37:07.072192 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:37:07.072420 3022 external_mini_cluster.cc:1428] Setting key 5e80d4b3b4a967da386a89bb9e915616
I20260620 11:37:07.073666 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.064909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:54417 (local address 127.2.243.190:33047)
0620 11:37:07.064960 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:07.064962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:07.065176 (+ 214us) server_negotiation.cc:407] Connection header received
0620 11:37:07.065693 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:07.065694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:07.065706 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:07.065788 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:07.066690 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.067250 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.068245 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.068443 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.068529 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.070584 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:07.070585 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:07.070586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:07.070593 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:07.072166 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:07.072692 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:07.072693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:07.072694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:07.072759 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:07.073036 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:07.073037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:07.073037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:07.073160 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:07.073248 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:07.073533 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 11:37:07.073602 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:37:07.073863 20508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.064868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33047 (local address 127.2.243.131:54417)
0620 11:37:07.065056 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:07.065089 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:07.065584 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:07.065858 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:07.065862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:07.066406 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:07.066587 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.066592 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.067350 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.067351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.068141 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.068143 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.068537 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.068538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.068669 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.069102 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:07.069110 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:07.070493 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:07.072270 (+ 1777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:07.072278 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:07.072280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:07.072579 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:07.072829 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:07.072830 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:07.072830 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:07.072966 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:07.073252 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:07.073259 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:07.073414 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 11:37:07.073500 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 11:37:07.073620 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":102,"threads_started":1}
I20260620 11:37:07.074342 20506 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33047
I20260620 11:37:07.074435 20506 heartbeater.cc:464] Registering TS with master...
I20260620 11:37:07.074572 20506 heartbeater.cc:511] Master 127.2.243.190:33047 requested a full tablet report, sending...
I20260620 11:37:07.074944 20047 ts_manager.cc:194] Registered new tserver with Master: 3c9a05e587804c8fa3c091f84ef9ab7c (127.2.243.131:34769)
I20260620 11:37:07.075335 20047 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:54417
I20260620 11:37:07.083986 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:37:07.091408 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.085385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53914 (local address 127.2.243.190:33047)
0620 11:37:07.085460 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:07.085463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:07.085574 (+ 111us) server_negotiation.cc:407] Connection header received
0620 11:37:07.085652 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:07.085653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:07.085662 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:07.085733 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:07.086351 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.086830 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.087654 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.087847 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.087930 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.089029 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:07.089031 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:07.089031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:07.089039 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:07.090160 (+ 1121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:07.090496 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:07.090497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:07.090498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:07.090553 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:07.090777 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:07.090778 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:07.090778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:07.091002 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:07.091108 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:07.091214 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 11:37:07.091310 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 11:37:07.092772 20047 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:37:07.094208 20047 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:37:07.100669 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.096248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54428 (local address 127.2.243.129:38709)
0620 11:37:07.096451 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:07.096457 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:07.096585 (+ 128us) server_negotiation.cc:407] Connection header received
0620 11:37:07.096665 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:07.096667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:07.096684 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:07.096778 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:07.096959 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.098030 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.099390 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.100108 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.100175 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.100236 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:07.100508 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 11:37:07.100554 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":94,"threads_started":1}
I20260620 11:37:07.100719 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.096301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51810 (local address 127.2.243.131:34769)
0620 11:37:07.096735 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:07.096739 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:07.096775 (+ 36us) server_negotiation.cc:407] Connection header received
0620 11:37:07.096809 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:07.096811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:07.096825 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:07.096907 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:07.097089 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.097890 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.099276 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.100076 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.100152 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.100222 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:07.100597 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 11:37:07.100636 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":73,"threads_started":1}
I20260620 11:37:07.100903 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.096148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:34769 (local address 127.0.0.1:51810)
0620 11:37:07.096468 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:07.096507 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:07.096587 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:07.096962 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:07.096963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:07.096965 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:07.097009 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.097011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.097998 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.097999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.099207 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.099210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.100262 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.100263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.100397 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.100400 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:37:07.100468 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 11:37:07.100529 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":63,"threads_started":1}
I20260620 11:37:07.101059 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.096363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:39409 (local address 127.0.0.1:50570)
0620 11:37:07.096678 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:07.096685 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:07.096721 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:07.097270 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:07.097271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:07.097272 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:07.097305 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.097307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.098571 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.098572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.099698 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.099700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.100674 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.100675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.100730 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.100735 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:37:07.100940 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 11:37:07.100985 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":54,"threads_started":1}
I20260620 11:37:07.100680 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.096143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:38709 (local address 127.0.0.1:54428)
0620 11:37:07.096467 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:07.096507 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:07.096586 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:07.096796 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:07.096799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:07.096801 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:07.096866 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.096868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.098150 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.098150 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.099308 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.099311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.100269 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.100269 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.100343 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.100374 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 11:37:07.100487 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 11:37:07.100531 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":62,"threads_started":1}
I20260620 11:37:07.101344 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.096526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50570 (local address 127.2.243.130:39409)
0620 11:37:07.096922 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:07.096927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:07.097003 (+ 76us) server_negotiation.cc:407] Connection header received
0620 11:37:07.097106 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:07.097109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:07.097129 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:07.097198 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:07.097388 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.098454 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.099787 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.100534 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.100593 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.100641 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:07.101198 (+ 557us) server_negotiation.cc:299] Negotiation successful
0620 11:37:07.101262 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":92,"threads_started":1}
I20260620 11:37:07.101698 20440 tablet_service.cc:1467] Processing CreateTablet for tablet c32a1a0eb50e4881bb683ff763c76521 (DEFAULT_TABLE table=test-table [id=8297663fd1a64b1c9c4c6ca1c91a5bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:37:07.101975 20440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32a1a0eb50e4881bb683ff763c76521. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:07.101920 20306 tablet_service.cc:1467] Processing CreateTablet for tablet c32a1a0eb50e4881bb683ff763c76521 (DEFAULT_TABLE table=test-table [id=8297663fd1a64b1c9c4c6ca1c91a5bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:37:07.102151 20306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32a1a0eb50e4881bb683ff763c76521. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:07.103621 20172 tablet_service.cc:1467] Processing CreateTablet for tablet c32a1a0eb50e4881bb683ff763c76521 (DEFAULT_TABLE table=test-table [id=8297663fd1a64b1c9c4c6ca1c91a5bff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:37:07.103951 20172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c32a1a0eb50e4881bb683ff763c76521. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:07.104759 20520 tablet_bootstrap.cc:492] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c: Bootstrap starting.
I20260620 11:37:07.105189 20521 tablet_bootstrap.cc:492] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6: Bootstrap starting.
I20260620 11:37:07.105671 20520 tablet_bootstrap.cc:654] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:07.105831 20521 tablet_bootstrap.cc:654] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:07.106288 20520 log.cc:824] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:07.106412 20521 log.cc:824] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:07.106807 20524 tablet_bootstrap.cc:492] T c32a1a0eb50e4881bb683ff763c76521 P 26103ea752f24a27aea1ae095b1d8425: Bootstrap starting.
I20260620 11:37:07.107200 20520 tablet_bootstrap.cc:492] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c: No bootstrap required, opened a new log
I20260620 11:37:07.107276 20520 ts_tablet_manager.cc:1397] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 11:37:07.107328 20521 tablet_bootstrap.cc:492] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6: No bootstrap required, opened a new log
I20260620 11:37:07.107410 20521 ts_tablet_manager.cc:1397] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:37:07.107533 20524 tablet_bootstrap.cc:654] T c32a1a0eb50e4881bb683ff763c76521 P 26103ea752f24a27aea1ae095b1d8425: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:07.108146 20524 log.cc:824] T c32a1a0eb50e4881bb683ff763c76521 P 26103ea752f24a27aea1ae095b1d8425: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:07.108974 20524 tablet_bootstrap.cc:492] T c32a1a0eb50e4881bb683ff763c76521 P 26103ea752f24a27aea1ae095b1d8425: No bootstrap required, opened a new log
I20260620 11:37:07.109041 20524 ts_tablet_manager.cc:1397] T c32a1a0eb50e4881bb683ff763c76521 P 26103ea752f24a27aea1ae095b1d8425: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 11:37:07.109210 20521 raft_consensus.cc:348] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26103ea752f24a27aea1ae095b1d8425" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38709 } } peers { permanent_uuid: "3c9a05e587804c8fa3c091f84ef9ab7c" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 34769 } } peers { permanent_uuid: "dcb274c89c31479f89e4b186f70648d6" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39409 } }
I20260620 11:37:07.109364 20521 raft_consensus.cc:374] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:07.109407 20521 raft_consensus.cc:729] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcb274c89c31479f89e4b186f70648d6, State: Initialized, Role: FOLLOWER
I20260620 11:37:07.109329 20520 raft_consensus.cc:348] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26103ea752f24a27aea1ae095b1d8425" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38709 } } peers { permanent_uuid: "3c9a05e587804c8fa3c091f84ef9ab7c" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 34769 } } peers { permanent_uuid: "dcb274c89c31479f89e4b186f70648d6" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39409 } }
I20260620 11:37:07.109493 20520 raft_consensus.cc:374] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:07.109524 20520 raft_consensus.cc:729] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9a05e587804c8fa3c091f84ef9ab7c, State: Initialized, Role: FOLLOWER
I20260620 11:37:07.109513 20521 consensus_queue.cc:260] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26103ea752f24a27aea1ae095b1d8425" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38709 } } peers { permanent_uuid: "3c9a05e587804c8fa3c091f84ef9ab7c" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 34769 } } peers { permanent_uuid: "dcb274c89c31479f89e4b186f70648d6" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39409 } }
I20260620 11:37:07.109640 20520 consensus_queue.cc:260] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26103ea752f24a27aea1ae095b1d8425" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38709 } } peers { permanent_uuid: "3c9a05e587804c8fa3c091f84ef9ab7c" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 34769 } } peers { permanent_uuid: "dcb274c89c31479f89e4b186f70648d6" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39409 } }
I20260620 11:37:07.109792 20521 ts_tablet_manager.cc:1428] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:37:07.109831 20372 heartbeater.cc:503] Master 127.2.243.190:33047 was elected leader, sending a full tablet report...
I20260620 11:37:07.109892 20520 ts_tablet_manager.cc:1428] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:37:07.110107 20506 heartbeater.cc:503] Master 127.2.243.190:33047 was elected leader, sending a full tablet report...
I20260620 11:37:07.111007 20524 raft_consensus.cc:348] T c32a1a0eb50e4881bb683ff763c76521 P 26103ea752f24a27aea1ae095b1d8425 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26103ea752f24a27aea1ae095b1d8425" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38709 } } peers { permanent_uuid: "3c9a05e587804c8fa3c091f84ef9ab7c" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 34769 } } peers { permanent_uuid: "dcb274c89c31479f89e4b186f70648d6" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39409 } }
I20260620 11:37:07.111176 20524 raft_consensus.cc:374] T c32a1a0eb50e4881bb683ff763c76521 P 26103ea752f24a27aea1ae095b1d8425 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:07.111217 20524 raft_consensus.cc:729] T c32a1a0eb50e4881bb683ff763c76521 P 26103ea752f24a27aea1ae095b1d8425 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26103ea752f24a27aea1ae095b1d8425, State: Initialized, Role: FOLLOWER
I20260620 11:37:07.111341 20524 consensus_queue.cc:260] T c32a1a0eb50e4881bb683ff763c76521 P 26103ea752f24a27aea1ae095b1d8425 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26103ea752f24a27aea1ae095b1d8425" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38709 } } peers { permanent_uuid: "3c9a05e587804c8fa3c091f84ef9ab7c" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 34769 } } peers { permanent_uuid: "dcb274c89c31479f89e4b186f70648d6" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39409 } }
I20260620 11:37:07.111604 20524 ts_tablet_manager.cc:1428] T c32a1a0eb50e4881bb683ff763c76521 P 26103ea752f24a27aea1ae095b1d8425: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 11:37:07.111618 20238 heartbeater.cc:503] Master 127.2.243.190:33047 was elected leader, sending a full tablet report...
W20260620 11:37:07.173949 20373 tablet.cc:2369] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 11:37:07.277159 20239 tablet.cc:2369] T c32a1a0eb50e4881bb683ff763c76521 P 26103ea752f24a27aea1ae095b1d8425: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:37:07.312137 20526 raft_consensus.cc:482] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:37:07.312258 20526 raft_consensus.cc:504] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26103ea752f24a27aea1ae095b1d8425" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38709 } } peers { permanent_uuid: "3c9a05e587804c8fa3c091f84ef9ab7c" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 34769 } } peers { permanent_uuid: "dcb274c89c31479f89e4b186f70648d6" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39409 } }
I20260620 11:37:07.312678 20526 leader_election.cc:288] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26103ea752f24a27aea1ae095b1d8425 (127.2.243.129:38709), 3c9a05e587804c8fa3c091f84ef9ab7c (127.2.243.131:34769)
W20260620 11:37:07.314805 20507 tablet.cc:2369] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:37:07.315569 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.312958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:52577 (local address 127.2.243.131:34769)
0620 11:37:07.313020 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:07.313022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:07.313031 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:37:07.313080 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:07.313081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:07.313089 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:07.313157 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:07.313327 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.313785 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.314650 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.315202 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.315307 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.315317 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:07.315472 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 11:37:07.315506 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 11:37:07.315569 20374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.312864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:34769 (local address 127.2.243.130:52577)
0620 11:37:07.312927 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:07.312941 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:07.313003 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:07.313160 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:07.313161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:07.313163 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:07.313249 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.313251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.313861 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.313862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.314559 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.314561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.315306 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.315307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.315388 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.315391 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:37:07.315464 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 11:37:07.315504 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 11:37:07.315717 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.312864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:38709 (local address 127.2.243.130:41257)
0620 11:37:07.313127 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:07.313135 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:07.313235 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:07.313396 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:07.313396 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:07.313398 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:07.313473 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.313475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.314064 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.314065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.314859 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.314862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.315465 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.315465 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.315517 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.315520 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:37:07.315584 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 11:37:07.315621 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":67,"thread_start_us":122,"threads_started":1}
I20260620 11:37:07.315712 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.312962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:41257 (local address 127.2.243.129:38709)
0620 11:37:07.313021 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:07.313024 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:07.313203 (+ 179us) server_negotiation.cc:407] Connection header received
0620 11:37:07.313314 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:07.313316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:07.313325 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:07.313394 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:07.313543 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.313981 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.314985 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.315374 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.315412 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.315418 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:07.315591 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 11:37:07.315623 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:37:07.315958 20460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32a1a0eb50e4881bb683ff763c76521" candidate_uuid: "dcb274c89c31479f89e4b186f70648d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9a05e587804c8fa3c091f84ef9ab7c" is_pre_election: true
I20260620 11:37:07.316095 20460 raft_consensus.cc:2454] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcb274c89c31479f89e4b186f70648d6 in term 0.
I20260620 11:37:07.316066 20528 raft_consensus.cc:482] T c32a1a0eb50e4881bb683ff763c76521 P 26103ea752f24a27aea1ae095b1d8425 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:37:07.316064 20192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32a1a0eb50e4881bb683ff763c76521" candidate_uuid: "dcb274c89c31479f89e4b186f70648d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26103ea752f24a27aea1ae095b1d8425" is_pre_election: true
I20260620 11:37:07.316252 20528 raft_consensus.cc:504] T c32a1a0eb50e4881bb683ff763c76521 P 26103ea752f24a27aea1ae095b1d8425 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26103ea752f24a27aea1ae095b1d8425" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38709 } } peers { permanent_uuid: "3c9a05e587804c8fa3c091f84ef9ab7c" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 34769 } } peers { permanent_uuid: "dcb274c89c31479f89e4b186f70648d6" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39409 } }
I20260620 11:37:07.316313 20262 leader_election.cc:302] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9a05e587804c8fa3c091f84ef9ab7c, dcb274c89c31479f89e4b186f70648d6; no voters:
I20260620 11:37:07.316390 20192 raft_consensus.cc:2454] T c32a1a0eb50e4881bb683ff763c76521 P 26103ea752f24a27aea1ae095b1d8425 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dcb274c89c31479f89e4b186f70648d6 in term 0.
I20260620 11:37:07.316426 20526 raft_consensus.cc:2781] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:37:07.316473 20526 raft_consensus.cc:482] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:37:07.316499 20526 raft_consensus.cc:3037] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:07.316555 20528 leader_election.cc:288] T c32a1a0eb50e4881bb683ff763c76521 P 26103ea752f24a27aea1ae095b1d8425 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c9a05e587804c8fa3c091f84ef9ab7c (127.2.243.131:34769), dcb274c89c31479f89e4b186f70648d6 (127.2.243.130:39409)
I20260620 11:37:07.317502 20526 raft_consensus.cc:504] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26103ea752f24a27aea1ae095b1d8425" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38709 } } peers { permanent_uuid: "3c9a05e587804c8fa3c091f84ef9ab7c" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 34769 } } peers { permanent_uuid: "dcb274c89c31479f89e4b186f70648d6" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39409 } }
I20260620 11:37:07.317732 20526 leader_election.cc:288] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6 [CANDIDATE]: Term 1 election: Requested vote from peers 26103ea752f24a27aea1ae095b1d8425 (127.2.243.129:38709), 3c9a05e587804c8fa3c091f84ef9ab7c (127.2.243.131:34769)
I20260620 11:37:07.317828 20460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32a1a0eb50e4881bb683ff763c76521" candidate_uuid: "dcb274c89c31479f89e4b186f70648d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9a05e587804c8fa3c091f84ef9ab7c"
I20260620 11:37:07.317904 20460 raft_consensus.cc:3037] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:07.317960 20192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32a1a0eb50e4881bb683ff763c76521" candidate_uuid: "dcb274c89c31479f89e4b186f70648d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26103ea752f24a27aea1ae095b1d8425"
I20260620 11:37:07.318035 20192 raft_consensus.cc:3037] T c32a1a0eb50e4881bb683ff763c76521 P 26103ea752f24a27aea1ae095b1d8425 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:07.318591 20460 raft_consensus.cc:2454] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcb274c89c31479f89e4b186f70648d6 in term 1.
I20260620 11:37:07.318754 20262 leader_election.cc:302] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c9a05e587804c8fa3c091f84ef9ab7c, dcb274c89c31479f89e4b186f70648d6; no voters:
I20260620 11:37:07.318904 20526 raft_consensus.cc:2781] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:07.318915 20192 raft_consensus.cc:2454] T c32a1a0eb50e4881bb683ff763c76521 P 26103ea752f24a27aea1ae095b1d8425 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dcb274c89c31479f89e4b186f70648d6 in term 1.
I20260620 11:37:07.319087 20526 raft_consensus.cc:686] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6 [term 1 LEADER]: Becoming Leader. State: Replica: dcb274c89c31479f89e4b186f70648d6, State: Running, Role: LEADER
I20260620 11:37:07.319370 20526 consensus_queue.cc:237] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26103ea752f24a27aea1ae095b1d8425" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38709 } } peers { permanent_uuid: "3c9a05e587804c8fa3c091f84ef9ab7c" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 34769 } } peers { permanent_uuid: "dcb274c89c31479f89e4b186f70648d6" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39409 } }
I20260620 11:37:07.319952 20517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.316730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:45941 (local address 127.2.243.131:34769)
0620 11:37:07.316794 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:07.316796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:07.316856 (+ 60us) server_negotiation.cc:407] Connection header received
0620 11:37:07.316959 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:07.316960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:07.316968 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:07.317023 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:07.317143 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.317791 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.319122 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.319571 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.319617 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.319624 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:07.319853 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 11:37:07.319890 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 11:37:07.320219 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.316674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:34769 (local address 127.2.243.129:45941)
0620 11:37:07.316796 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:07.316806 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:07.316881 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:07.317027 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:07.317028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:07.317029 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:07.317084 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.317085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.317870 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.317871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.319027 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.319029 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.319685 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.319685 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.319767 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.319771 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:37:07.320069 (+ 298us) client_negotiation.cc:240] Negotiation successful
0620 11:37:07.320113 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":6,"thread_start_us":52,"threads_started":1}
I20260620 11:37:07.320125 20047 catalog_manager.cc:5495] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to dcb274c89c31479f89e4b186f70648d6 (127.2.243.130). New cstate: current_term: 1 leader_uuid: "dcb274c89c31479f89e4b186f70648d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26103ea752f24a27aea1ae095b1d8425" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3c9a05e587804c8fa3c091f84ef9ab7c" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 34769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcb274c89c31479f89e4b186f70648d6" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39409 } health_report { overall_health: HEALTHY } } }
I20260620 11:37:07.320447 20460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32a1a0eb50e4881bb683ff763c76521" candidate_uuid: "26103ea752f24a27aea1ae095b1d8425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c9a05e587804c8fa3c091f84ef9ab7c" is_pre_election: true
I20260620 11:37:07.320533 20460 raft_consensus.cc:2379] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 26103ea752f24a27aea1ae095b1d8425 in current term 1: Already voted for candidate dcb274c89c31479f89e4b186f70648d6 in this term.
I20260620 11:37:07.320871 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.316821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:39409 (local address 127.2.243.129:59841)
0620 11:37:07.316951 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:07.316961 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:07.317040 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:07.317189 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:07.317189 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:07.317191 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:07.317249 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.317250 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.318792 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.318793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.319882 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.319884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.320533 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.320534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.320602 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.320605 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:37:07.320745 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 11:37:07.320790 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":34,"threads_started":1}
I20260620 11:37:07.320977 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.316866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:59841 (local address 127.2.243.130:39409)
0620 11:37:07.316899 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:07.316901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:07.317016 (+ 115us) server_negotiation.cc:407] Connection header received
0620 11:37:07.317095 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:07.317096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:07.317105 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:07.317155 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:07.317326 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.317870 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.319974 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.320413 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.320471 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.320478 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:07.320861 (+ 383us) server_negotiation.cc:299] Negotiation successful
0620 11:37:07.320903 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:37:07.321170 20326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c32a1a0eb50e4881bb683ff763c76521" candidate_uuid: "26103ea752f24a27aea1ae095b1d8425" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcb274c89c31479f89e4b186f70648d6" is_pre_election: true
I20260620 11:37:07.321434 20128 leader_election.cc:302] T c32a1a0eb50e4881bb683ff763c76521 P 26103ea752f24a27aea1ae095b1d8425 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26103ea752f24a27aea1ae095b1d8425; no voters: 3c9a05e587804c8fa3c091f84ef9ab7c, dcb274c89c31479f89e4b186f70648d6
I20260620 11:37:07.321548 20528 raft_consensus.cc:2726] T c32a1a0eb50e4881bb683ff763c76521 P 26103ea752f24a27aea1ae095b1d8425 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 11:37:07.331092 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.329026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50574 (local address 127.2.243.130:39409)
0620 11:37:07.329088 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:07.329090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:07.329097 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:37:07.329122 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:07.329123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:07.329136 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:07.329192 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:07.329315 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.329727 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.330389 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.330562 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.330607 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.330734 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:07.330830 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:07.330940 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:07.330961 (+ 21us) server_negotiation.cc:299] Negotiation successful
0620 11:37:07.330978 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 11:37:07.332624 20460 raft_consensus.cc:1261] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c [term 1 FOLLOWER]: Refusing update from remote peer dcb274c89c31479f89e4b186f70648d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:37:07.332643 20192 raft_consensus.cc:1261] T c32a1a0eb50e4881bb683ff763c76521 P 26103ea752f24a27aea1ae095b1d8425 [term 1 FOLLOWER]: Refusing update from remote peer dcb274c89c31479f89e4b186f70648d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:37:07.332931 20526 consensus_queue.cc:1035] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c9a05e587804c8fa3c091f84ef9ab7c" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 34769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:37:07.333041 20532 consensus_queue.cc:1035] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26103ea752f24a27aea1ae095b1d8425" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:37:07.335338 20535 mvcc.cc:205] Tried to move back new op lower bound from 7298889430352285696 to 7298889430300864512. Current Snapshot: MvccSnapshot[applied={T|T < 7298889430352285696}]
I20260620 11:37:07.336283 20536 mvcc.cc:205] Tried to move back new op lower bound from 7298889430352285696 to 7298889430300864512. Current Snapshot: MvccSnapshot[applied={T|T < 7298889430352285696 or (T in {7298889430352285696})}]
I20260620 11:37:07.337576 20046 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:37:07.337666 20046 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:37:07.338367 20046 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 240ac80952724e9f97ec1f8715b98af4: Sending DeleteTablet for 3 replicas of tablet c32a1a0eb50e4881bb683ff763c76521
I20260620 11:37:07.338672 20172 tablet_service.cc:1514] Processing DeleteTablet for tablet c32a1a0eb50e4881bb683ff763c76521 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:37:07 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:54428
I20260620 11:37:07.338708 20440 tablet_service.cc:1514] Processing DeleteTablet for tablet c32a1a0eb50e4881bb683ff763c76521 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:37:07 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:51810
I20260620 11:37:07.338699 20306 tablet_service.cc:1514] Processing DeleteTablet for tablet c32a1a0eb50e4881bb683ff763c76521 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:37:07 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:50570
I20260620 11:37:07.338938 20546 tablet_replica.cc:330] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6: stopping tablet replica
I20260620 11:37:07.339030 20546 raft_consensus.cc:2229] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:37:07.339130 20546 raft_consensus.cc:2258] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:37:07.339270 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 20109
I20260620 11:37:07.339291 20545 tablet_replica.cc:330] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c: stopping tablet replica
I20260620 11:37:07.339368 20545 raft_consensus.cc:2229] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:37:07.339447 20545 raft_consensus.cc:2258] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:37:07.340548 20546 ts_tablet_manager.cc:1905] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:37:07.341272 20545 ts_tablet_manager.cc:1905] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:37:07.342604 20546 ts_tablet_manager.cc:1918] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:37:07.342765 20546 log.cc:1196] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-1/wal/wals/c32a1a0eb50e4881bb683ff763c76521
I20260620 11:37:07.342830 20545 ts_tablet_manager.cc:1918] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:37:07.342942 20545 log.cc:1196] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/c32a1a0eb50e4881bb683ff763c76521
I20260620 11:37:07.343214 20546 ts_tablet_manager.cc:1939] T c32a1a0eb50e4881bb683ff763c76521 P dcb274c89c31479f89e4b186f70648d6: Deleting consensus metadata
I20260620 11:37:07.343243 20545 ts_tablet_manager.cc:1939] T c32a1a0eb50e4881bb683ff763c76521 P 3c9a05e587804c8fa3c091f84ef9ab7c: Deleting consensus metadata
I20260620 11:37:07.343629 20034 catalog_manager.cc:4841] TS dcb274c89c31479f89e4b186f70648d6 (127.2.243.130:39409): tablet c32a1a0eb50e4881bb683ff763c76521 (table test-table [id=8297663fd1a64b1c9c4c6ca1c91a5bff]) successfully deleted
I20260620 11:37:07.344014 20034 catalog_manager.cc:4841] TS 3c9a05e587804c8fa3c091f84ef9ab7c (127.2.243.131:34769): tablet c32a1a0eb50e4881bb683ff763c76521 (table test-table [id=8297663fd1a64b1c9c4c6ca1c91a5bff]) successfully deleted
W20260620 11:37:07.345569 20033 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.243.129:38709 (error 108)
I20260620 11:37:07.345788 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 20243
I20260620 11:37:07.346402 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.345919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:38709 (local address 127.0.0.1:54444)
0620 11:37:07.346328 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:07.346372 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.129:38709: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386}
W20260620 11:37:07.346513 20033 catalog_manager.cc:4584] TS 26103ea752f24a27aea1ae095b1d8425 (127.2.243.129:38709): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c32a1a0eb50e4881bb683ff763c76521: Network error: Client connection negotiation failed: client connection to 127.2.243.129:38709: connect: Connection refused (error 111)
I20260620 11:37:07.350693 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 20377
I20260620 11:37:07.355188 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 20015
2026-06-20T11:37:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2043 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T11:37:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:07Z Disabled control of system clock
I20260620 11:37:07.382201 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:33031
--webserver_interface=127.2.243.190
--webserver_port=0
--builtin_ntp_servers=127.2.243.148:37249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:33031
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 11:37:07.461205 20551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:07.461400 20551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:07.461419 20551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:07.462586 20551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:07.462627 20551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:07.462639 20551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:07.462652 20551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:07.462706 20551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:07.464010 20551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:37249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:33031
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:33031
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:07.464275 20551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:07.464551 20551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:07.469548 20557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:07.469507 20556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:07.469511 20559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:07.469713 20551 server_base.cc:1034] running on GCE node
I20260620 11:37:07.469957 20551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:07.470240 20551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:07.471395 20551 hybrid_clock.cc:648] HybridClock initialized: now 1781955427471375 us; error 42 us; skew 500 ppm
I20260620 11:37:07.472581 20551 webserver.cc:460] Webserver started at http://127.2.243.190:46237/ using document root <none> and password file <none>
I20260620 11:37:07.472819 20551 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:07.472887 20551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:07.473028 20551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:07.474079 20551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "02287bad2ceb4f3e8019318fa61bf460"
format_stamp: "Formatted at 2026-06-20 11:37:07 on dist-test-slave-ctbw"
server_key: "dc56aaad530768135741456b9c884e4e"
server_key_iv: "45245d60c1491174d85ec506af796fde"
server_key_version: "encryptionkey@0"
I20260620 11:37:07.474432 20551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "02287bad2ceb4f3e8019318fa61bf460"
format_stamp: "Formatted at 2026-06-20 11:37:07 on dist-test-slave-ctbw"
server_key: "dc56aaad530768135741456b9c884e4e"
server_key_iv: "45245d60c1491174d85ec506af796fde"
server_key_version: "encryptionkey@0"
I20260620 11:37:07.475834 20551 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:37:07.476555 20565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:07.476717 20551 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:37:07.476828 20551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "02287bad2ceb4f3e8019318fa61bf460"
format_stamp: "Formatted at 2026-06-20 11:37:07 on dist-test-slave-ctbw"
server_key: "dc56aaad530768135741456b9c884e4e"
server_key_iv: "45245d60c1491174d85ec506af796fde"
server_key_version: "encryptionkey@0"
I20260620 11:37:07.476898 20551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:07.494670 20551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:07.495033 20551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:07.495186 20551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:07.499081 20551 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:33031
I20260620 11:37:07.499114 20617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:33031 every 8 connection(s)
I20260620 11:37:07.499450 20551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:07.500411 20618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:07.502676 20618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460: Bootstrap starting.
I20260620 11:37:07.503325 20618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:07.503615 20618 log.cc:824] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:07.504287 20618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460: No bootstrap required, opened a new log
I20260620 11:37:07.505834 20618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02287bad2ceb4f3e8019318fa61bf460" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33031 } }
I20260620 11:37:07.505967 20618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:07.505990 20618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02287bad2ceb4f3e8019318fa61bf460, State: Initialized, Role: FOLLOWER
I20260620 11:37:07.506161 20618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02287bad2ceb4f3e8019318fa61bf460" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33031 } }
I20260620 11:37:07.506229 20618 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:07.506251 20618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:07.506315 20618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:07.506505 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 20551
I20260620 11:37:07.506611 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:07.506790 3022 external_mini_cluster.cc:1428] Setting key f67c8087792d42397d6b6f41b6a26464
I20260620 11:37:07.506929 20618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02287bad2ceb4f3e8019318fa61bf460" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33031 } }
I20260620 11:37:07.507048 20618 leader_election.cc:302] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02287bad2ceb4f3e8019318fa61bf460; no voters:
I20260620 11:37:07.507211 20618 leader_election.cc:288] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:07.507267 20621 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:07.507423 20621 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460 [term 1 LEADER]: Becoming Leader. State: Replica: 02287bad2ceb4f3e8019318fa61bf460, State: Running, Role: LEADER
I20260620 11:37:07.507483 20618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:07.507512 20621 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02287bad2ceb4f3e8019318fa61bf460" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33031 } }
I20260620 11:37:07.507826 20621 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02287bad2ceb4f3e8019318fa61bf460. Latest consensus state: current_term: 1 leader_uuid: "02287bad2ceb4f3e8019318fa61bf460" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02287bad2ceb4f3e8019318fa61bf460" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33031 } } }
I20260620 11:37:07.507915 20621 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:07.507943 20625 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02287bad2ceb4f3e8019318fa61bf460" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02287bad2ceb4f3e8019318fa61bf460" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33031 } } }
I20260620 11:37:07.508005 20625 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:07.508745 20629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:07.509311 20629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:07.511480 20629 catalog_manager.cc:1347] Generated new cluster ID: 8280ae9fd2ea419183fc2909af663a94
I20260620 11:37:07.511534 20629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:37:07.514184 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.507286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42446 (local address 127.2.243.190:33031)
0620 11:37:07.508898 (+ 1612us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:07.508907 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:07.509115 (+ 208us) server_negotiation.cc:407] Connection header received
0620 11:37:07.509270 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:07.509275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:07.509328 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:07.509489 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:07.509747 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.510974 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.512050 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.512665 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.512810 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.513462 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:07.513466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:07.513467 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:07.513675 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:07.513716 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:07.513740 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:07.513814 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:07.513893 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 11:37:07.513972 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1538,"thread_start_us":76,"threads_started":1}
I20260620 11:37:07.518435 20629 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:07.518967 20629 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:07.526439 20629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460: Generated new TSK 0
I20260620 11:37:07.526598 20629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:37:07.531715 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33031
--builtin_ntp_servers=127.2.243.148:37249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 11:37:07.618664 20642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:07.618880 20642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:07.618933 20642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:07.620093 20642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:07.620131 20642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:07.620157 20642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:37:07.621469 20642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:37249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:07.621701 20642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:07.622004 20642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:07.627741 20647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:07.627767 20650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:07.627837 20648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:07.628074 20642 server_base.cc:1034] running on GCE node
I20260620 11:37:07.628262 20642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:07.628484 20642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:07.629647 20642 hybrid_clock.cc:648] HybridClock initialized: now 1781955427629631 us; error 33 us; skew 500 ppm
I20260620 11:37:07.630945 20642 webserver.cc:460] Webserver started at http://127.2.243.129:37493/ using document root <none> and password file <none>
I20260620 11:37:07.631197 20642 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:07.631265 20642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:07.631404 20642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:07.632402 20642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "b6d5dd31bb4f436daac7cacec145ce39"
format_stamp: "Formatted at 2026-06-20 11:37:07 on dist-test-slave-ctbw"
server_key: "309265e5987d23b8735738244c8c6a95"
server_key_iv: "1b4bcb249e24e4bd2771398bbcd27648"
server_key_version: "encryptionkey@0"
I20260620 11:37:07.632752 20642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "b6d5dd31bb4f436daac7cacec145ce39"
format_stamp: "Formatted at 2026-06-20 11:37:07 on dist-test-slave-ctbw"
server_key: "309265e5987d23b8735738244c8c6a95"
server_key_iv: "1b4bcb249e24e4bd2771398bbcd27648"
server_key_version: "encryptionkey@0"
I20260620 11:37:07.634019 20642 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:37:07.634747 20656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:07.634972 20642 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:37:07.635070 20642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "b6d5dd31bb4f436daac7cacec145ce39"
format_stamp: "Formatted at 2026-06-20 11:37:07 on dist-test-slave-ctbw"
server_key: "309265e5987d23b8735738244c8c6a95"
server_key_iv: "1b4bcb249e24e4bd2771398bbcd27648"
server_key_version: "encryptionkey@0"
I20260620 11:37:07.635145 20642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:07.664090 20642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:07.664350 20642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:07.664451 20642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:07.664630 20642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:37:07.664916 20642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:37:07.664947 20642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:07.664968 20642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:37:07.664980 20642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:07.670501 20642 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:45349
I20260620 11:37:07.670581 20769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:45349 every 8 connection(s)
I20260620 11:37:07.671010 20642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:37:07.675895 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.671712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:45713 (local address 127.2.243.190:33031)
0620 11:37:07.671802 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:07.671804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:07.672008 (+ 204us) server_negotiation.cc:407] Connection header received
0620 11:37:07.672472 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:07.672473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:07.672483 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:07.672539 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:07.673173 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.673646 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.674574 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.674762 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.674840 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.675689 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:07.675690 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:07.675691 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:07.675696 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:07.675710 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:07.675713 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:07.675760 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:07.675819 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 11:37:07.675838 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 11:37:07.676100 20772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.671691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33031 (local address 127.2.243.129:45713)
0620 11:37:07.671884 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:07.671916 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:07.672381 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:07.672591 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:07.672594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:07.672602 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:07.673097 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.673102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.673722 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.673723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.674497 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.674499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.674832 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.674833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.674998 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.675561 (+ 563us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:37:07.675561 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:07.675590 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:37:07.675611 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:07.675758 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:07.675760 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:07.675762 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:37:07.675808 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 11:37:07.675879 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":110,"threads_started":1}
I20260620 11:37:07.676555 20770 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33031
I20260620 11:37:07.676652 20770 heartbeater.cc:464] Registering TS with master...
I20260620 11:37:07.676820 20770 heartbeater.cc:511] Master 127.2.243.190:33031 requested a full tablet report, sending...
I20260620 11:37:07.677315 20582 ts_manager.cc:194] Registered new tserver with Master: b6d5dd31bb4f436daac7cacec145ce39 (127.2.243.129:45349)
I20260620 11:37:07.677595 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 20642
I20260620 11:37:07.677699 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:37:07.677883 3022 external_mini_cluster.cc:1428] Setting key 1ab84fcfb2570992597d120e66a640bf
I20260620 11:37:07.678238 20582 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.243.129:45713
I20260620 11:37:07.678906 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33031
--builtin_ntp_servers=127.2.243.148:37249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 11:37:07.757484 20773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:07.757663 20773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:07.757684 20773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:07.758766 20773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:07.758800 20773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:07.758823 20773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:37:07.760197 20773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:37249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:07.760380 20773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:07.760614 20773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:07.765233 20779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:07.765303 20781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:07.765272 20778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:07.765614 20773 server_base.cc:1034] running on GCE node
I20260620 11:37:07.765847 20773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:07.766075 20773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:07.767246 20773 hybrid_clock.cc:648] HybridClock initialized: now 1781955427767231 us; error 25 us; skew 500 ppm
I20260620 11:37:07.768390 20773 webserver.cc:460] Webserver started at http://127.2.243.130:33189/ using document root <none> and password file <none>
I20260620 11:37:07.768640 20773 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:07.768708 20773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:07.768862 20773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:07.769850 20773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "309f5c295ffa4c548eb65d037767c151"
format_stamp: "Formatted at 2026-06-20 11:37:07 on dist-test-slave-ctbw"
server_key: "61b6035c9a35476e1e96713f82a20103"
server_key_iv: "f1a81a655cbaff9e567b9e7af7e0fb6b"
server_key_version: "encryptionkey@0"
I20260620 11:37:07.770191 20773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "309f5c295ffa4c548eb65d037767c151"
format_stamp: "Formatted at 2026-06-20 11:37:07 on dist-test-slave-ctbw"
server_key: "61b6035c9a35476e1e96713f82a20103"
server_key_iv: "f1a81a655cbaff9e567b9e7af7e0fb6b"
server_key_version: "encryptionkey@0"
I20260620 11:37:07.771520 20773 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:37:07.772212 20787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:07.772373 20773 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:37:07.772476 20773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "309f5c295ffa4c548eb65d037767c151"
format_stamp: "Formatted at 2026-06-20 11:37:07 on dist-test-slave-ctbw"
server_key: "61b6035c9a35476e1e96713f82a20103"
server_key_iv: "f1a81a655cbaff9e567b9e7af7e0fb6b"
server_key_version: "encryptionkey@0"
I20260620 11:37:07.772547 20773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:07.778738 20773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:07.779045 20773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:07.779188 20773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:07.779399 20773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:37:07.779688 20773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:37:07.779744 20773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:07.779778 20773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:37:07.779808 20773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:07.785359 20773 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:37473
I20260620 11:37:07.785425 20900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:37473 every 8 connection(s)
I20260620 11:37:07.785745 20773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:37:07.790817 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.786467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:48097 (local address 127.2.243.190:33031)
0620 11:37:07.786537 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:07.786538 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:07.786735 (+ 197us) server_negotiation.cc:407] Connection header received
0620 11:37:07.787291 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:07.787293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:07.787306 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:07.787368 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:07.788106 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.788597 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.789519 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.789712 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.789778 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.790579 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:07.790581 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:07.790581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:07.790585 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:07.790599 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:07.790604 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:07.790655 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:07.790735 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 11:37:07.790760 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 11:37:07.791113 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.786436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33031 (local address 127.2.243.130:48097)
0620 11:37:07.786609 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:07.786646 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:07.787157 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:07.787440 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:07.787443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:07.787452 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:07.788014 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.788019 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.788677 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.788678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.789441 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.789443 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.789784 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.789785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.789886 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.790430 (+ 544us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:37:07.790431 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:07.790458 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:37:07.790478 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:07.790668 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:07.790672 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:07.790674 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:37:07.790725 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 11:37:07.790795 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260620 11:37:07.791585 20901 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33031
I20260620 11:37:07.791683 20901 heartbeater.cc:464] Registering TS with master...
I20260620 11:37:07.791864 20901 heartbeater.cc:511] Master 127.2.243.190:33031 requested a full tablet report, sending...
I20260620 11:37:07.792212 20581 ts_manager.cc:194] Registered new tserver with Master: 309f5c295ffa4c548eb65d037767c151 (127.2.243.130:37473)
I20260620 11:37:07.792589 20581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.243.130:48097
I20260620 11:37:07.793463 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 20773
I20260620 11:37:07.793563 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:37:07.793773 3022 external_mini_cluster.cc:1428] Setting key 4b9c2976b01f6d4434bc5b15a8882b29
I20260620 11:37:07.794569 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33031
--builtin_ntp_servers=127.2.243.148:37249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 11:37:07.873292 20904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:07.873471 20904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:07.873492 20904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:07.874661 20904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:07.874699 20904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:07.874744 20904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:37:07.876235 20904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:37249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33031
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:07.876461 20904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:07.876740 20904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:07.881500 20912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:07.881529 20909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:07.881625 20910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:07.881906 20904 server_base.cc:1034] running on GCE node
I20260620 11:37:07.882120 20904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:07.882373 20904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:07.883535 20904 hybrid_clock.cc:648] HybridClock initialized: now 1781955427883519 us; error 46 us; skew 500 ppm
I20260620 11:37:07.884744 20904 webserver.cc:460] Webserver started at http://127.2.243.131:44299/ using document root <none> and password file <none>
I20260620 11:37:07.885008 20904 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:07.885079 20904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:07.885241 20904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:07.886287 20904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "abf2cf2f77e64044b8a708ed691ab820"
format_stamp: "Formatted at 2026-06-20 11:37:07 on dist-test-slave-ctbw"
server_key: "418472395b21dd33407e20e19146ca9b"
server_key_iv: "22bae39e3c26c02a907571122b84a768"
server_key_version: "encryptionkey@0"
I20260620 11:37:07.886631 20904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "abf2cf2f77e64044b8a708ed691ab820"
format_stamp: "Formatted at 2026-06-20 11:37:07 on dist-test-slave-ctbw"
server_key: "418472395b21dd33407e20e19146ca9b"
server_key_iv: "22bae39e3c26c02a907571122b84a768"
server_key_version: "encryptionkey@0"
I20260620 11:37:07.888012 20904 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:37:07.888759 20918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:07.888922 20904 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:37:07.889029 20904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "abf2cf2f77e64044b8a708ed691ab820"
format_stamp: "Formatted at 2026-06-20 11:37:07 on dist-test-slave-ctbw"
server_key: "418472395b21dd33407e20e19146ca9b"
server_key_iv: "22bae39e3c26c02a907571122b84a768"
server_key_version: "encryptionkey@0"
I20260620 11:37:07.889098 20904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:07.904326 20904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:07.904600 20904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:07.904742 20904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:07.904968 20904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:37:07.905279 20904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:37:07.905333 20904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:07.905372 20904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:37:07.905400 20904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:07.910997 20904 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:42531
I20260620 11:37:07.911032 21031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:42531 every 8 connection(s)
I20260620 11:37:07.911363 20904 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:37:07.916638 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.912166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:34117 (local address 127.2.243.190:33031)
0620 11:37:07.912254 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:07.912256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:07.912507 (+ 251us) server_negotiation.cc:407] Connection header received
0620 11:37:07.913018 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:07.913019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:07.913029 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:07.913067 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:07.913751 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.914231 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.915292 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.915477 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.915524 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.916427 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:07.916428 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:07.916428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:07.916431 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:07.916441 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:07.916444 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:07.916481 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:07.916565 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 11:37:07.916582 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 11:37:07.916890 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.912152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33031 (local address 127.2.243.131:34117)
0620 11:37:07.912386 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:07.912421 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:07.912914 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:07.913137 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:07.913140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:07.913148 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:07.913678 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.913683 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.914321 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.914322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.915211 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.915214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.915559 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.915560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.915684 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.916302 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:37:07.916303 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:07.916331 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:37:07.916355 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:07.916498 (+ 143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:07.916502 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:07.916504 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:37:07.916559 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 11:37:07.916640 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":121,"threads_started":1}
I20260620 11:37:07.917399 21032 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33031
I20260620 11:37:07.917490 21032 heartbeater.cc:464] Registering TS with master...
I20260620 11:37:07.917639 21032 heartbeater.cc:511] Master 127.2.243.190:33031 requested a full tablet report, sending...
I20260620 11:37:07.918012 20581 ts_manager.cc:194] Registered new tserver with Master: abf2cf2f77e64044b8a708ed691ab820 (127.2.243.131:42531)
I20260620 11:37:07.918383 20581 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.243.131:34117
I20260620 11:37:07.918883 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 20904
I20260620 11:37:07.918998 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:37:07.919221 3022 external_mini_cluster.cc:1428] Setting key 6bae5813710bf7196a540acbbb6ce0b1
I20260620 11:37:07.920046 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:37:07.924234 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.921303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42460 (local address 127.2.243.190:33031)
0620 11:37:07.921361 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:07.921363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:07.921547 (+ 184us) server_negotiation.cc:407] Connection header received
0620 11:37:07.921608 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:07.921608 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:07.921618 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:07.921677 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:07.921802 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.922277 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.923043 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.923252 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.923313 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.923924 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:07.923925 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:07.923925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:07.923932 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:07.923943 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:07.923946 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:07.923991 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:07.924104 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 11:37:07.924122 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 11:37:07.925897 20581 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42460:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:37:07.927568 20581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:37:07.933852 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.929903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36302 (local address 127.2.243.131:42531)
0620 11:37:07.930018 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:07.930021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:07.930600 (+ 579us) server_negotiation.cc:407] Connection header received
0620 11:37:07.930640 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:07.930642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:07.930656 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:07.930723 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:07.930827 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.931511 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.933017 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.933485 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.933571 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.933664 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:07.933739 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 11:37:07.933773 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":65,"threads_started":1}
I20260620 11:37:07.933841 21040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.929663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:42531 (local address 127.0.0.1:36302)
0620 11:37:07.930397 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:07.930406 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:07.930452 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:07.930732 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:07.930732 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:07.930734 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:07.930777 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.930778 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.931609 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.931610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.932939 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.932940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.933571 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.933572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.933642 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.933654 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 11:37:07.933727 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 11:37:07.933774 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":98,"threads_started":1}
I20260620 11:37:07.934058 21041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.929657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:45349 (local address 127.0.0.1:53510)
0620 11:37:07.930267 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:07.930274 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:07.930329 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:07.930564 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:07.930565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:07.930568 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:07.930610 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.930611 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.931908 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.931909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.932757 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.932759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.933701 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.933703 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.933762 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.933766 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:37:07.933796 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 11:37:07.933836 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":99,"threads_started":1}
I20260620 11:37:07.934119 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.929882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53510 (local address 127.2.243.129:45349)
0620 11:37:07.930049 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:07.930054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:07.930336 (+ 282us) server_negotiation.cc:407] Connection header received
0620 11:37:07.930391 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:07.930394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:07.930412 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:07.930490 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:07.930689 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.931805 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.932841 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.933592 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.933882 (+ 290us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.933974 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:07.933998 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 11:37:07.934034 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260620 11:37:07.934119 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.929737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50052 (local address 127.2.243.130:37473)
0620 11:37:07.929900 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:07.929904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:07.930194 (+ 290us) server_negotiation.cc:407] Connection header received
0620 11:37:07.930251 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:07.930252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:07.930265 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:07.930358 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:07.930586 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.931774 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.932986 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.933668 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.933739 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.933826 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:07.933997 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 11:37:07.934036 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260620 11:37:07.934458 21042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:07.929653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:37473 (local address 127.0.0.1:50052)
0620 11:37:07.930079 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:07.930114 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:07.930182 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:07.930363 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:07.930366 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:07.930368 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:07.930493 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.930496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.931861 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.931862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.932903 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:07.932905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:07.933857 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:07.933858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:07.933907 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:07.933909 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:37:07.934378 (+ 469us) client_negotiation.cc:240] Negotiation successful
0620 11:37:07.934412 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":55,"threads_started":1}
I20260620 11:37:07.934829 20704 tablet_service.cc:1467] Processing CreateTablet for tablet e9092ece5abd4f17a11987eb898d8531 (DEFAULT_TABLE table=test-table [id=413a76b1b4e046d79a0bb5cab9adc185]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:37:07.935235 20704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9092ece5abd4f17a11987eb898d8531. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:07.935246 20835 tablet_service.cc:1467] Processing CreateTablet for tablet e9092ece5abd4f17a11987eb898d8531 (DEFAULT_TABLE table=test-table [id=413a76b1b4e046d79a0bb5cab9adc185]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:37:07.935551 20835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9092ece5abd4f17a11987eb898d8531. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:07.938583 21047 tablet_bootstrap.cc:492] T e9092ece5abd4f17a11987eb898d8531 P 309f5c295ffa4c548eb65d037767c151: Bootstrap starting.
I20260620 11:37:07.938849 20966 tablet_service.cc:1467] Processing CreateTablet for tablet e9092ece5abd4f17a11987eb898d8531 (DEFAULT_TABLE table=test-table [id=413a76b1b4e046d79a0bb5cab9adc185]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:37:07.939105 21046 tablet_bootstrap.cc:492] T e9092ece5abd4f17a11987eb898d8531 P b6d5dd31bb4f436daac7cacec145ce39: Bootstrap starting.
I20260620 11:37:07.939138 20966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9092ece5abd4f17a11987eb898d8531. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:07.939293 21047 tablet_bootstrap.cc:654] T e9092ece5abd4f17a11987eb898d8531 P 309f5c295ffa4c548eb65d037767c151: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:07.939692 21047 log.cc:824] T e9092ece5abd4f17a11987eb898d8531 P 309f5c295ffa4c548eb65d037767c151: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:07.939894 21046 tablet_bootstrap.cc:654] T e9092ece5abd4f17a11987eb898d8531 P b6d5dd31bb4f436daac7cacec145ce39: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:07.940145 21046 log.cc:824] T e9092ece5abd4f17a11987eb898d8531 P b6d5dd31bb4f436daac7cacec145ce39: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:07.940567 21047 tablet_bootstrap.cc:492] T e9092ece5abd4f17a11987eb898d8531 P 309f5c295ffa4c548eb65d037767c151: No bootstrap required, opened a new log
I20260620 11:37:07.940655 21047 ts_tablet_manager.cc:1397] T e9092ece5abd4f17a11987eb898d8531 P 309f5c295ffa4c548eb65d037767c151: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 11:37:07.940985 21046 tablet_bootstrap.cc:492] T e9092ece5abd4f17a11987eb898d8531 P b6d5dd31bb4f436daac7cacec145ce39: No bootstrap required, opened a new log
I20260620 11:37:07.941052 21046 ts_tablet_manager.cc:1397] T e9092ece5abd4f17a11987eb898d8531 P b6d5dd31bb4f436daac7cacec145ce39: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:37:07.941957 21050 tablet_bootstrap.cc:492] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820: Bootstrap starting.
I20260620 11:37:07.942507 21050 tablet_bootstrap.cc:654] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:07.942792 21050 log.cc:824] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:07.943418 21050 tablet_bootstrap.cc:492] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820: No bootstrap required, opened a new log
I20260620 11:37:07.943493 21050 ts_tablet_manager.cc:1397] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:37:07.943385 21047 raft_consensus.cc:348] T e9092ece5abd4f17a11987eb898d8531 P 309f5c295ffa4c548eb65d037767c151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d5dd31bb4f436daac7cacec145ce39" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45349 } } peers { permanent_uuid: "309f5c295ffa4c548eb65d037767c151" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37473 } } peers { permanent_uuid: "abf2cf2f77e64044b8a708ed691ab820" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 42531 } }
I20260620 11:37:07.943537 21047 raft_consensus.cc:374] T e9092ece5abd4f17a11987eb898d8531 P 309f5c295ffa4c548eb65d037767c151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:07.943588 21047 raft_consensus.cc:729] T e9092ece5abd4f17a11987eb898d8531 P 309f5c295ffa4c548eb65d037767c151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309f5c295ffa4c548eb65d037767c151, State: Initialized, Role: FOLLOWER
I20260620 11:37:07.943554 21046 raft_consensus.cc:348] T e9092ece5abd4f17a11987eb898d8531 P b6d5dd31bb4f436daac7cacec145ce39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d5dd31bb4f436daac7cacec145ce39" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45349 } } peers { permanent_uuid: "309f5c295ffa4c548eb65d037767c151" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37473 } } peers { permanent_uuid: "abf2cf2f77e64044b8a708ed691ab820" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 42531 } }
I20260620 11:37:07.943701 21046 raft_consensus.cc:374] T e9092ece5abd4f17a11987eb898d8531 P b6d5dd31bb4f436daac7cacec145ce39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:07.943744 21046 raft_consensus.cc:729] T e9092ece5abd4f17a11987eb898d8531 P b6d5dd31bb4f436daac7cacec145ce39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6d5dd31bb4f436daac7cacec145ce39, State: Initialized, Role: FOLLOWER
I20260620 11:37:07.943732 21047 consensus_queue.cc:260] T e9092ece5abd4f17a11987eb898d8531 P 309f5c295ffa4c548eb65d037767c151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d5dd31bb4f436daac7cacec145ce39" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45349 } } peers { permanent_uuid: "309f5c295ffa4c548eb65d037767c151" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37473 } } peers { permanent_uuid: "abf2cf2f77e64044b8a708ed691ab820" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 42531 } }
I20260620 11:37:07.943867 21046 consensus_queue.cc:260] T e9092ece5abd4f17a11987eb898d8531 P b6d5dd31bb4f436daac7cacec145ce39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d5dd31bb4f436daac7cacec145ce39" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45349 } } peers { permanent_uuid: "309f5c295ffa4c548eb65d037767c151" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37473 } } peers { permanent_uuid: "abf2cf2f77e64044b8a708ed691ab820" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 42531 } }
I20260620 11:37:07.943997 21047 ts_tablet_manager.cc:1428] T e9092ece5abd4f17a11987eb898d8531 P 309f5c295ffa4c548eb65d037767c151: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 11:37:07.944090 21046 ts_tablet_manager.cc:1428] T e9092ece5abd4f17a11987eb898d8531 P b6d5dd31bb4f436daac7cacec145ce39: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:37:07.944182 20901 heartbeater.cc:503] Master 127.2.243.190:33031 was elected leader, sending a full tablet report...
I20260620 11:37:07.944209 20770 heartbeater.cc:503] Master 127.2.243.190:33031 was elected leader, sending a full tablet report...
I20260620 11:37:07.945881 21050 raft_consensus.cc:348] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d5dd31bb4f436daac7cacec145ce39" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45349 } } peers { permanent_uuid: "309f5c295ffa4c548eb65d037767c151" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37473 } } peers { permanent_uuid: "abf2cf2f77e64044b8a708ed691ab820" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 42531 } }
I20260620 11:37:07.946022 21050 raft_consensus.cc:374] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:07.946064 21050 raft_consensus.cc:729] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abf2cf2f77e64044b8a708ed691ab820, State: Initialized, Role: FOLLOWER
I20260620 11:37:07.946184 21050 consensus_queue.cc:260] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d5dd31bb4f436daac7cacec145ce39" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45349 } } peers { permanent_uuid: "309f5c295ffa4c548eb65d037767c151" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37473 } } peers { permanent_uuid: "abf2cf2f77e64044b8a708ed691ab820" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 42531 } }
I20260620 11:37:07.946408 21050 ts_tablet_manager.cc:1428] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:37:07.946476 21032 heartbeater.cc:503] Master 127.2.243.190:33031 was elected leader, sending a full tablet report...
W20260620 11:37:08.036274 20902 tablet.cc:2369] T e9092ece5abd4f17a11987eb898d8531 P 309f5c295ffa4c548eb65d037767c151: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:37:08.056661 21054 raft_consensus.cc:482] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:37:08.056818 21054 raft_consensus.cc:504] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d5dd31bb4f436daac7cacec145ce39" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45349 } } peers { permanent_uuid: "309f5c295ffa4c548eb65d037767c151" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37473 } } peers { permanent_uuid: "abf2cf2f77e64044b8a708ed691ab820" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 42531 } }
I20260620 11:37:08.057215 21054 leader_election.cc:288] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6d5dd31bb4f436daac7cacec145ce39 (127.2.243.129:45349), 309f5c295ffa4c548eb65d037767c151 (127.2.243.130:37473)
I20260620 11:37:08.059965 21044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:08.057430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:47295 (local address 127.2.243.129:45349)
0620 11:37:08.057473 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:08.057475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:08.057484 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:37:08.057576 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:08.057577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:08.057590 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:08.057674 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:08.057829 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:08.058300 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:08.059252 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:08.059606 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:08.059692 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:08.059700 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:08.059858 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 11:37:08.059897 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:37:08.059964 21034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:08.057379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:45349 (local address 127.2.243.131:47295)
0620 11:37:08.057440 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:08.057452 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:08.057496 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:08.057677 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:08.057678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:08.057680 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:08.057755 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:08.057757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:08.058382 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:08.058384 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:08.059158 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:08.059160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:08.059699 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:08.059699 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:08.059785 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:08.059788 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:37:08.059853 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 11:37:08.059894 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 11:37:08.060111 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:08.057478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:58605 (local address 127.2.243.130:37473)
0620 11:37:08.057524 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:08.057526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:08.057682 (+ 156us) server_negotiation.cc:407] Connection header received
0620 11:37:08.057750 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:08.057751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:08.057761 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:08.057829 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:08.057980 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:08.058431 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:08.059401 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:08.059758 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:08.059839 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:08.059847 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:08.060009 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 11:37:08.060047 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:37:08.060120 21055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:08.057397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:37473 (local address 127.2.243.131:58605)
0620 11:37:08.057595 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:08.057604 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:08.057683 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:08.057833 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:08.057833 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:08.057835 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:08.057908 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:08.057909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:08.058515 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:08.058517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:08.059309 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:08.059312 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:08.059853 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:08.059854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:08.059928 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:08.059931 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:37:08.060002 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 11:37:08.060042 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":125,"threads_started":1}
I20260620 11:37:08.060313 20724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9092ece5abd4f17a11987eb898d8531" candidate_uuid: "abf2cf2f77e64044b8a708ed691ab820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d5dd31bb4f436daac7cacec145ce39" is_pre_election: true
I20260620 11:37:08.060477 20724 raft_consensus.cc:2454] T e9092ece5abd4f17a11987eb898d8531 P b6d5dd31bb4f436daac7cacec145ce39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abf2cf2f77e64044b8a708ed691ab820 in term 0.
I20260620 11:37:08.060443 20855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9092ece5abd4f17a11987eb898d8531" candidate_uuid: "abf2cf2f77e64044b8a708ed691ab820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309f5c295ffa4c548eb65d037767c151" is_pre_election: true
I20260620 11:37:08.060563 20855 raft_consensus.cc:2454] T e9092ece5abd4f17a11987eb898d8531 P 309f5c295ffa4c548eb65d037767c151 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abf2cf2f77e64044b8a708ed691ab820 in term 0.
I20260620 11:37:08.060756 20921 leader_election.cc:302] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: abf2cf2f77e64044b8a708ed691ab820, b6d5dd31bb4f436daac7cacec145ce39; no voters:
I20260620 11:37:08.060895 21054 raft_consensus.cc:2781] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:37:08.060952 21054 raft_consensus.cc:482] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:37:08.060999 21054 raft_consensus.cc:3037] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:08.061686 21054 raft_consensus.cc:504] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d5dd31bb4f436daac7cacec145ce39" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45349 } } peers { permanent_uuid: "309f5c295ffa4c548eb65d037767c151" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37473 } } peers { permanent_uuid: "abf2cf2f77e64044b8a708ed691ab820" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 42531 } }
I20260620 11:37:08.061828 21054 leader_election.cc:288] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820 [CANDIDATE]: Term 1 election: Requested vote from peers b6d5dd31bb4f436daac7cacec145ce39 (127.2.243.129:45349), 309f5c295ffa4c548eb65d037767c151 (127.2.243.130:37473)
I20260620 11:37:08.061967 20855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9092ece5abd4f17a11987eb898d8531" candidate_uuid: "abf2cf2f77e64044b8a708ed691ab820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309f5c295ffa4c548eb65d037767c151"
I20260620 11:37:08.061983 20724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9092ece5abd4f17a11987eb898d8531" candidate_uuid: "abf2cf2f77e64044b8a708ed691ab820" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6d5dd31bb4f436daac7cacec145ce39"
I20260620 11:37:08.062049 20855 raft_consensus.cc:3037] T e9092ece5abd4f17a11987eb898d8531 P 309f5c295ffa4c548eb65d037767c151 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:08.062054 20724 raft_consensus.cc:3037] T e9092ece5abd4f17a11987eb898d8531 P b6d5dd31bb4f436daac7cacec145ce39 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:08.062803 20855 raft_consensus.cc:2454] T e9092ece5abd4f17a11987eb898d8531 P 309f5c295ffa4c548eb65d037767c151 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abf2cf2f77e64044b8a708ed691ab820 in term 1.
I20260620 11:37:08.063045 20724 raft_consensus.cc:2454] T e9092ece5abd4f17a11987eb898d8531 P b6d5dd31bb4f436daac7cacec145ce39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abf2cf2f77e64044b8a708ed691ab820 in term 1.
I20260620 11:37:08.063067 20919 leader_election.cc:302] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 309f5c295ffa4c548eb65d037767c151, abf2cf2f77e64044b8a708ed691ab820; no voters:
I20260620 11:37:08.063181 21054 raft_consensus.cc:2781] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:08.063367 21054 raft_consensus.cc:686] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820 [term 1 LEADER]: Becoming Leader. State: Replica: abf2cf2f77e64044b8a708ed691ab820, State: Running, Role: LEADER
I20260620 11:37:08.063476 21054 consensus_queue.cc:237] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d5dd31bb4f436daac7cacec145ce39" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45349 } } peers { permanent_uuid: "309f5c295ffa4c548eb65d037767c151" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37473 } } peers { permanent_uuid: "abf2cf2f77e64044b8a708ed691ab820" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 42531 } }
I20260620 11:37:08.064292 20581 catalog_manager.cc:5495] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820 reported cstate change: term changed from 0 to 1, leader changed from <none> to abf2cf2f77e64044b8a708ed691ab820 (127.2.243.131). New cstate: current_term: 1 leader_uuid: "abf2cf2f77e64044b8a708ed691ab820" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b6d5dd31bb4f436daac7cacec145ce39" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "309f5c295ffa4c548eb65d037767c151" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abf2cf2f77e64044b8a708ed691ab820" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 42531 } health_report { overall_health: HEALTHY } } }
I20260620 11:37:08.077304 21045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:08.074841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36312 (local address 127.2.243.131:42531)
0620 11:37:08.074925 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:08.074927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:08.074943 (+ 16us) server_negotiation.cc:407] Connection header received
0620 11:37:08.075003 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:08.075004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:08.075019 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:08.075079 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:08.075281 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:08.075777 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:08.076610 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:08.076795 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:08.076847 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:08.076993 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:08.077086 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:08.077146 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:08.077219 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 11:37:08.077234 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 11:37:08.079174 20855 raft_consensus.cc:1261] T e9092ece5abd4f17a11987eb898d8531 P 309f5c295ffa4c548eb65d037767c151 [term 1 FOLLOWER]: Refusing update from remote peer abf2cf2f77e64044b8a708ed691ab820: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:37:08.079191 20724 raft_consensus.cc:1261] T e9092ece5abd4f17a11987eb898d8531 P b6d5dd31bb4f436daac7cacec145ce39 [term 1 FOLLOWER]: Refusing update from remote peer abf2cf2f77e64044b8a708ed691ab820: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:37:08.079551 21056 consensus_queue.cc:1035] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309f5c295ffa4c548eb65d037767c151" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 37473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:37:08.079686 21054 consensus_queue.cc:1035] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6d5dd31bb4f436daac7cacec145ce39" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 45349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:37:08.081895 21059 mvcc.cc:205] Tried to move back new op lower bound from 7298889433409409024 to 7298889433348898816. Current Snapshot: MvccSnapshot[applied={T|T < 7298889433409409024}]
I20260620 11:37:08.082269 21061 mvcc.cc:205] Tried to move back new op lower bound from 7298889433409409024 to 7298889433348898816. Current Snapshot: MvccSnapshot[applied={T|T < 7298889433409409024}]
I20260620 11:37:08.082472 21060 mvcc.cc:205] Tried to move back new op lower bound from 7298889433409409024 to 7298889433348898816. Current Snapshot: MvccSnapshot[applied={T|T < 7298889433409409024}]
I20260620 11:37:08.084511 20581 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:37:08.084590 20581 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42460:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:37:08.085376 20581 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 02287bad2ceb4f3e8019318fa61bf460: Sending DeleteTablet for 3 replicas of tablet e9092ece5abd4f17a11987eb898d8531
I20260620 11:37:08.085695 20704 tablet_service.cc:1514] Processing DeleteTablet for tablet e9092ece5abd4f17a11987eb898d8531 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:37:08 UTC) from {username='slave'} at 127.0.0.1:53510
I20260620 11:37:08.085708 20835 tablet_service.cc:1514] Processing DeleteTablet for tablet e9092ece5abd4f17a11987eb898d8531 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:37:08 UTC) from {username='slave'} at 127.0.0.1:50052
I20260620 11:37:08.085781 20966 tablet_service.cc:1514] Processing DeleteTablet for tablet e9092ece5abd4f17a11987eb898d8531 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:37:08 UTC) from {username='slave'} at 127.0.0.1:36302
I20260620 11:37:08.085976 21071 tablet_replica.cc:330] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820: stopping tablet replica
I20260620 11:37:08.086045 21071 raft_consensus.cc:2229] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:37:08.086128 21071 raft_consensus.cc:2258] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:37:08.086365 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 20642
I20260620 11:37:08.086516 21071 ts_tablet_manager.cc:1905] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:37:08.088089 21071 ts_tablet_manager.cc:1918] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:37:08.088142 21071 log.cc:1196] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/e9092ece5abd4f17a11987eb898d8531
I20260620 11:37:08.088421 21071 ts_tablet_manager.cc:1939] T e9092ece5abd4f17a11987eb898d8531 P abf2cf2f77e64044b8a708ed691ab820: Deleting consensus metadata
I20260620 11:37:08.088735 20567 catalog_manager.cc:4841] TS abf2cf2f77e64044b8a708ed691ab820 (127.2.243.131:42531): tablet e9092ece5abd4f17a11987eb898d8531 (table test-table [id=413a76b1b4e046d79a0bb5cab9adc185]) successfully deleted
I20260620 11:37:08.091012 21070 tablet_replica.cc:330] T e9092ece5abd4f17a11987eb898d8531 P 309f5c295ffa4c548eb65d037767c151: stopping tablet replica
I20260620 11:37:08.091128 21070 raft_consensus.cc:2229] T e9092ece5abd4f17a11987eb898d8531 P 309f5c295ffa4c548eb65d037767c151 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:37:08.091230 21070 raft_consensus.cc:2258] T e9092ece5abd4f17a11987eb898d8531 P 309f5c295ffa4c548eb65d037767c151 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:37:08.091578 21070 ts_tablet_manager.cc:1905] T e9092ece5abd4f17a11987eb898d8531 P 309f5c295ffa4c548eb65d037767c151: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 11:37:08.092427 20568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.243.129:45349 (error 108)
I20260620 11:37:08.092861 21042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:08.092743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:45349 (local address 127.0.0.1:53514)
0620 11:37:08.092787 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:08.092826 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.129:45349: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 11:37:08.092924 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 20773
I20260620 11:37:08.093034 21070 ts_tablet_manager.cc:1918] T e9092ece5abd4f17a11987eb898d8531 P 309f5c295ffa4c548eb65d037767c151: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260620 11:37:08.093542 20568 catalog_manager.cc:4584] TS b6d5dd31bb4f436daac7cacec145ce39 (127.2.243.129:45349): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e9092ece5abd4f17a11987eb898d8531: Network error: Client connection negotiation failed: client connection to 127.2.243.129:45349: connect: Connection refused (error 111)
I20260620 11:37:08.099114 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 20904
I20260620 11:37:08.099148 21042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:08.099020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:37473 (local address 127.0.0.1:50054)
0620 11:37:08.099078 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:08.099110 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.130:37473: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 11:37:08.103142 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 20551
2026-06-20T11:37:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (747 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:08 dist-test-slave-ctbw krb5kdc[21075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:08 dist-test-slave-ctbw krb5kdc[21075](info): set up 2 sockets
Jun 20 11:37:08 dist-test-slave-ctbw krb5kdc[21075](info): commencing operation
krb5kdc: starting...
W20260620 11:37:09.134966 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:09 dist-test-slave-ctbw krb5kdc[21075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:37:09.250367 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:33731
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:43771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:33731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:09.330631 21088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:09.330889 21088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:09.330948 21088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:09.332060 21088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:09.332139 21088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:09.332163 21088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:09.332183 21088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:09.332209 21088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:09.333678 21088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:43771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:33731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:33731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:09.333961 21088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:09.334211 21088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:09.339030 21094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:09.339082 21096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:09.339030 21093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:09.339337 21088 server_base.cc:1034] running on GCE node
I20260620 11:37:09.339551 21088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:09.339875 21088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:09.341017 21088 hybrid_clock.cc:648] HybridClock initialized: now 1781955429340995 us; error 38 us; skew 500 ppm
Jun 20 11:37:09 dist-test-slave-ctbw krb5kdc[21075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955429, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:09.343560 21088 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:37:09.344033 21088 webserver.cc:460] Webserver started at http://127.2.243.190:39075/ using document root <none> and password file <none>
I20260620 11:37:09.344301 21088 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:09.344377 21088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:09.344522 21088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:09.345563 21088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "0fe2219c855b42e88ff1f3e7b48dee5e"
format_stamp: "Formatted at 2026-06-20 11:37:09 on dist-test-slave-ctbw"
server_key: "8e82fa81ef73bad975db181385030395"
server_key_iv: "b0d0023171bebe39f679053e9017aa29"
server_key_version: "encryptionkey@0"
I20260620 11:37:09.345927 21088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "0fe2219c855b42e88ff1f3e7b48dee5e"
format_stamp: "Formatted at 2026-06-20 11:37:09 on dist-test-slave-ctbw"
server_key: "8e82fa81ef73bad975db181385030395"
server_key_iv: "b0d0023171bebe39f679053e9017aa29"
server_key_version: "encryptionkey@0"
I20260620 11:37:09.347327 21088 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 11:37:09.348201 21103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:09.348423 21088 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:37:09.348526 21088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "0fe2219c855b42e88ff1f3e7b48dee5e"
format_stamp: "Formatted at 2026-06-20 11:37:09 on dist-test-slave-ctbw"
server_key: "8e82fa81ef73bad975db181385030395"
server_key_iv: "b0d0023171bebe39f679053e9017aa29"
server_key_version: "encryptionkey@0"
I20260620 11:37:09.348614 21088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:09.370368 21088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:09.372406 21088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:09.372574 21088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:09.376664 21088 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:33731
I20260620 11:37:09.376703 21155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:33731 every 8 connection(s)
I20260620 11:37:09.377087 21088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:09.377956 21156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:09.380235 21156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e: Bootstrap starting.
I20260620 11:37:09.380745 21156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:09.380966 21156 log.cc:824] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:09.381619 21156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e: No bootstrap required, opened a new log
I20260620 11:37:09.383194 21156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe2219c855b42e88ff1f3e7b48dee5e" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33731 } }
I20260620 11:37:09.383325 21156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:09.383347 21156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fe2219c855b42e88ff1f3e7b48dee5e, State: Initialized, Role: FOLLOWER
I20260620 11:37:09.383446 21156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe2219c855b42e88ff1f3e7b48dee5e" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33731 } }
I20260620 11:37:09.383498 21156 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:09.383518 21156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:09.383543 21156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:09.383982 21156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe2219c855b42e88ff1f3e7b48dee5e" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33731 } }
I20260620 11:37:09.384064 21156 leader_election.cc:302] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fe2219c855b42e88ff1f3e7b48dee5e; no voters:
I20260620 11:37:09.384186 21156 leader_election.cc:288] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:09.384246 21159 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:09.384460 21159 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e [term 1 LEADER]: Becoming Leader. State: Replica: 0fe2219c855b42e88ff1f3e7b48dee5e, State: Running, Role: LEADER
I20260620 11:37:09.384469 21156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:09.384572 21159 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe2219c855b42e88ff1f3e7b48dee5e" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33731 } }
I20260620 11:37:09.385046 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 21088
I20260620 11:37:09.385174 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:09.385121 21160 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fe2219c855b42e88ff1f3e7b48dee5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe2219c855b42e88ff1f3e7b48dee5e" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33731 } } }
I20260620 11:37:09.385202 21160 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:09.385368 3022 external_mini_cluster.cc:1428] Setting key a4a8d0abc55990f35ff13239af2929bf
I20260620 11:37:09.385644 21168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:09.385829 21161 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fe2219c855b42e88ff1f3e7b48dee5e. Latest consensus state: current_term: 1 leader_uuid: "0fe2219c855b42e88ff1f3e7b48dee5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fe2219c855b42e88ff1f3e7b48dee5e" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 33731 } } }
I20260620 11:37:09.385941 21161 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:09.386179 21168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:09.388541 21168 catalog_manager.cc:1347] Generated new cluster ID: 408a8be3eb7d47d8b3428a4ed5cc3775
I20260620 11:37:09.388619 21168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:37:09 dist-test-slave-ctbw krb5kdc[21075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:37:09.397053 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:09.386014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60126 (local address 127.2.243.190:33731)
0620 11:37:09.386189 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:09.386198 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:09.386235 (+ 37us) server_negotiation.cc:407] Connection header received
0620 11:37:09.386351 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:09.386359 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:09.386421 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:09.386602 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:09.387418 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.388427 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.389524 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.390185 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.390972 (+ 787us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:09.393175 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:09.393179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:09.393179 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:09.393190 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:09.395281 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:09.395696 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:09.395699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:09.395699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:09.395811 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:09.396034 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:09.396034 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:09.396035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:09.396329 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:09.396477 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:09.396615 (+ 138us) server_negotiation.cc:299] Negotiation successful
0620 11:37:09.396839 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260620 11:37:09.403669 21168 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:09.404191 21168 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:09.410271 21168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e: Generated new TSK 0
I20260620 11:37:09.410444 21168 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
I20260620 11:37:09.441960 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33731
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:43771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:09.522451 21182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:09.522643 21182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:09.522666 21182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:09.523759 21182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:09.523797 21182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:09.523823 21182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:37:09.525795 21182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:43771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:09.526053 21182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:09.526279 21182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:09.531177 21190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:09.531165 21187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:09.531275 21182 server_base.cc:1034] running on GCE node
W20260620 11:37:09.531379 21188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:09.531698 21182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:09.531895 21182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:09.533039 21182 hybrid_clock.cc:648] HybridClock initialized: now 1781955429533024 us; error 39 us; skew 500 ppm
Jun 20 11:37:09 dist-test-slave-ctbw krb5kdc[21075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955429, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:09.535612 21182 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:37:09.536031 21182 webserver.cc:460] Webserver started at http://127.2.243.129:40651/ using document root <none> and password file <none>
I20260620 11:37:09.536288 21182 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:09.536362 21182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:09.536509 21182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:09.537525 21182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "b697a0ee76ff4b578ef0fff06518577b"
format_stamp: "Formatted at 2026-06-20 11:37:09 on dist-test-slave-ctbw"
server_key: "d012b65309f77c2391c49658544c99b0"
server_key_iv: "7354651f2fa72855df29c5e2afc71a71"
server_key_version: "encryptionkey@0"
I20260620 11:37:09.537894 21182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "b697a0ee76ff4b578ef0fff06518577b"
format_stamp: "Formatted at 2026-06-20 11:37:09 on dist-test-slave-ctbw"
server_key: "d012b65309f77c2391c49658544c99b0"
server_key_iv: "7354651f2fa72855df29c5e2afc71a71"
server_key_version: "encryptionkey@0"
I20260620 11:37:09.539255 21182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 11:37:09.539978 21197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:09.540187 21182 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:37:09.540287 21182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "b697a0ee76ff4b578ef0fff06518577b"
format_stamp: "Formatted at 2026-06-20 11:37:09 on dist-test-slave-ctbw"
server_key: "d012b65309f77c2391c49658544c99b0"
server_key_iv: "7354651f2fa72855df29c5e2afc71a71"
server_key_version: "encryptionkey@0"
I20260620 11:37:09.540369 21182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:09.554009 21182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:09.555914 21182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:09.556087 21182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:09.556314 21182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:37:09.556622 21182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:37:09.556686 21182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:09.556744 21182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:37:09.556793 21182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:09.563539 21182 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:33441
I20260620 11:37:09.563594 21310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:33441 every 8 connection(s)
I20260620 11:37:09.563890 21182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:37:09.566599 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 21182
I20260620 11:37:09.566690 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:37:09.566936 3022 external_mini_cluster.cc:1428] Setting key fa389c7923dd5609bbeebc727e66b39a
Jun 20 11:37:09 dist-test-slave-ctbw krb5kdc[21075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955429, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:37:09.574002 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:09.564553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:33937 (local address 127.2.243.190:33731)
0620 11:37:09.564640 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:09.564643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:09.564855 (+ 212us) server_negotiation.cc:407] Connection header received
0620 11:37:09.565376 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:09.565377 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:09.565388 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:09.565447 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:09.566371 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.567097 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.568205 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.568426 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.568509 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:09.571078 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:09.571079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:09.571079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:09.571087 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:09.572576 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:09.573022 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:09.573024 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:09.573025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:09.573083 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:09.573348 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:09.573349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:09.573349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:09.573467 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:09.573543 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:09.573827 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 11:37:09.573911 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 11:37:09.574221 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:09.564536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33731 (local address 127.2.243.129:33937)
0620 11:37:09.564729 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:09.564763 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:09.565266 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:09.565520 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:09.565523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:09.566077 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:09.566264 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:09.566268 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.567197 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.567199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:09.568099 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:09.568102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.568516 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.568517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:09.568653 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:09.569329 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:09.569339 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:09.570982 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:09.572691 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:09.572698 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:09.572700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:09.572926 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:09.573158 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:09.573159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:09.573160 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:09.573273 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:09.573545 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:09.573552 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:09.573692 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 11:37:09.573804 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 11:37:09.573964 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":94,"threads_started":1}
WARNING: no policy specified for kudu/127.2.243.130@KRBTEST.COM; defaulting to no policy
I20260620 11:37:09.574697 21311 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33731
I20260620 11:37:09.574790 21311 heartbeater.cc:464] Registering TS with master...
I20260620 11:37:09.574995 21311 heartbeater.cc:511] Master 127.2.243.190:33731 requested a full tablet report, sending...
I20260620 11:37:09.575560 21120 ts_manager.cc:194] Registered new tserver with Master: b697a0ee76ff4b578ef0fff06518577b (127.2.243.129:33441)
I20260620 11:37:09.576246 21120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:33937
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
I20260620 11:37:09.592301 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33731
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:43771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:09.670751 21316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:09.671007 21316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:09.671046 21316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:09.672191 21316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:09.672264 21316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:09.672310 21316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:37:09.673751 21316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:43771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:09.674022 21316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:09.674258 21316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:09.679104 21322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:09.679113 21324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:09.679113 21321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:09.679327 21316 server_base.cc:1034] running on GCE node
I20260620 11:37:09.679508 21316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:09.679745 21316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:09.680913 21316 hybrid_clock.cc:648] HybridClock initialized: now 1781955429680903 us; error 29 us; skew 500 ppm
Jun 20 11:37:09 dist-test-slave-ctbw krb5kdc[21075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955429, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:09.683462 21316 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:37:09.683871 21316 webserver.cc:460] Webserver started at http://127.2.243.130:39377/ using document root <none> and password file <none>
I20260620 11:37:09.684154 21316 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:09.684233 21316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:09.684381 21316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:09.685416 21316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "a265b775d19944b78cbabd7ba25f8dcc"
format_stamp: "Formatted at 2026-06-20 11:37:09 on dist-test-slave-ctbw"
server_key: "6fcf266f139283a1ef292ac5cc50fd3f"
server_key_iv: "cb8538c1bf8b22e4f65ae05ab361479c"
server_key_version: "encryptionkey@0"
I20260620 11:37:09.685813 21316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "a265b775d19944b78cbabd7ba25f8dcc"
format_stamp: "Formatted at 2026-06-20 11:37:09 on dist-test-slave-ctbw"
server_key: "6fcf266f139283a1ef292ac5cc50fd3f"
server_key_iv: "cb8538c1bf8b22e4f65ae05ab361479c"
server_key_version: "encryptionkey@0"
I20260620 11:37:09.687216 21316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:37:09.687930 21331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:09.688138 21316 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:37:09.688238 21316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "a265b775d19944b78cbabd7ba25f8dcc"
format_stamp: "Formatted at 2026-06-20 11:37:09 on dist-test-slave-ctbw"
server_key: "6fcf266f139283a1ef292ac5cc50fd3f"
server_key_iv: "cb8538c1bf8b22e4f65ae05ab361479c"
server_key_version: "encryptionkey@0"
I20260620 11:37:09.688319 21316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:09.696895 21316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:09.698678 21316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:09.698839 21316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:09.699098 21316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:37:09.699399 21316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:37:09.699457 21316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:09.699492 21316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:37:09.699519 21316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:09.705403 21316 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:46695
I20260620 11:37:09.705521 21444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:46695 every 8 connection(s)
I20260620 11:37:09.705807 21316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:37:09.706619 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 21316
I20260620 11:37:09.706734 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:37:09.706970 3022 external_mini_cluster.cc:1428] Setting key 45e50c4539b8a98bc50300efe67ad715
Jun 20 11:37:09 dist-test-slave-ctbw krb5kdc[21075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955429, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.243.131@KRBTEST.COM; defaulting to no policy
I20260620 11:37:09.716512 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:09.706532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:49409 (local address 127.2.243.190:33731)
0620 11:37:09.706619 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:09.706621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:09.706840 (+ 219us) server_negotiation.cc:407] Connection header received
0620 11:37:09.707471 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:09.707472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:09.707482 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:09.707551 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:09.708876 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.709589 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.710584 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.710953 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.711032 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:09.713490 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:09.713492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:09.713492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:09.713500 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:09.715159 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:09.715548 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:09.715549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:09.715550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:09.715612 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:09.715861 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:09.715862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:09.715862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:09.716008 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:09.716086 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:09.716365 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 11:37:09.716442 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 11:37:09.716650 21447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:09.706481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33731 (local address 127.2.243.130:49409)
0620 11:37:09.706663 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:09.706742 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:09.707370 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:09.707644 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:09.707649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:09.708503 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:09.708774 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:09.708781 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.709689 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.709689 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:09.710490 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:09.710491 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.711055 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.711056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:09.711161 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:09.711773 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:09.711782 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:09.713380 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:09.715242 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:09.715248 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:09.715250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:09.715458 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:09.715690 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:09.715690 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:09.715691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:09.715788 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:09.716088 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:09.716094 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:09.716226 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 11:37:09.716313 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 11:37:09.716431 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":101,"threads_started":1}
I20260620 11:37:09.717276 21445 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33731
I20260620 11:37:09.717360 21445 heartbeater.cc:464] Registering TS with master...
I20260620 11:37:09.717558 21445 heartbeater.cc:511] Master 127.2.243.190:33731 requested a full tablet report, sending...
I20260620 11:37:09.717998 21120 ts_manager.cc:194] Registered new tserver with Master: a265b775d19944b78cbabd7ba25f8dcc (127.2.243.130:46695)
I20260620 11:37:09.718436 21120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:49409
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
I20260620 11:37:09.733352 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33731
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:43771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:09.813443 21450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:09.813643 21450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:09.813666 21450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:09.814916 21450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:09.814956 21450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:09.814983 21450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:37:09.816403 21450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:43771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:33731
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:09.816661 21450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:09.816885 21450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:09.821854 21455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:09.821859 21458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:09.821894 21456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:09.822132 21450 server_base.cc:1034] running on GCE node
I20260620 11:37:09.822320 21450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:09.822585 21450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:09.823736 21450 hybrid_clock.cc:648] HybridClock initialized: now 1781955429823715 us; error 37 us; skew 500 ppm
Jun 20 11:37:09 dist-test-slave-ctbw krb5kdc[21075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955429, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:09.826261 21450 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:37:09.826622 21450 webserver.cc:460] Webserver started at http://127.2.243.131:39057/ using document root <none> and password file <none>
I20260620 11:37:09.826807 21450 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:09.826846 21450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:09.827040 21450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:09.828012 21450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "3fc590aff8614380a34b2fb573a9f459"
format_stamp: "Formatted at 2026-06-20 11:37:09 on dist-test-slave-ctbw"
server_key: "a6c759a0d3212590acbe01f586fb8c7e"
server_key_iv: "c00a887ad1eeaf015f459e6907d620f4"
server_key_version: "encryptionkey@0"
I20260620 11:37:09.828296 21450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "3fc590aff8614380a34b2fb573a9f459"
format_stamp: "Formatted at 2026-06-20 11:37:09 on dist-test-slave-ctbw"
server_key: "a6c759a0d3212590acbe01f586fb8c7e"
server_key_iv: "c00a887ad1eeaf015f459e6907d620f4"
server_key_version: "encryptionkey@0"
I20260620 11:37:09.829479 21450 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:37:09.830154 21465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:09.830307 21450 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:37:09.830385 21450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "3fc590aff8614380a34b2fb573a9f459"
format_stamp: "Formatted at 2026-06-20 11:37:09 on dist-test-slave-ctbw"
server_key: "a6c759a0d3212590acbe01f586fb8c7e"
server_key_iv: "c00a887ad1eeaf015f459e6907d620f4"
server_key_version: "encryptionkey@0"
I20260620 11:37:09.830438 21450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:09.863454 21450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:09.865970 21450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:09.866091 21450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:09.866278 21450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:37:09.866619 21450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:37:09.866657 21450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:09.866679 21450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:37:09.866722 21450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:09.872718 21450 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:37353
I20260620 11:37:09.872761 21578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:37353 every 8 connection(s)
I20260620 11:37:09.873080 21450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:37:09.878350 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 21450
I20260620 11:37:09.878476 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:37:09.878625 3022 external_mini_cluster.cc:1428] Setting key 8ced738af90b0fba86942bdfacd1a654
Jun 20 11:37:09 dist-test-slave-ctbw krb5kdc[21075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955429, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:37:09.882284 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:09.873730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:37255 (local address 127.2.243.190:33731)
0620 11:37:09.873790 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:09.873792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:09.874000 (+ 208us) server_negotiation.cc:407] Connection header received
0620 11:37:09.874509 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:09.874511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:09.874523 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:09.874580 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:09.875536 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.876047 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.877044 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.877242 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.877335 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:09.879525 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:09.879527 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:09.879527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:09.879534 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:09.881067 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:09.881429 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:09.881430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:09.881430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:09.881478 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:09.881745 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:09.881745 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:09.881745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:09.881843 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:09.881893 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:09.882163 (+ 270us) server_negotiation.cc:299] Negotiation successful
0620 11:37:09.882222 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 11:37:09.882507 21581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:09.873723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:33731 (local address 127.2.243.131:37255)
0620 11:37:09.873862 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:09.873905 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:09.874395 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:09.874650 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:09.874654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:09.875232 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:09.875418 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:09.875422 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.876151 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.876152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:09.876941 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:09.876943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.877344 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.877345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:09.877468 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:09.877922 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:09.877930 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:09.879426 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:09.881136 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:09.881141 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:09.881142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:09.881347 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:09.881573 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:09.881574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:09.881575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:09.881683 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:09.881897 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:09.881901 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:09.882050 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 11:37:09.882132 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 11:37:09.882252 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":64,"threads_started":1}
I20260620 11:37:09.883054 21579 heartbeater.cc:347] Connected to a master server at 127.2.243.190:33731
I20260620 11:37:09.883132 21579 heartbeater.cc:464] Registering TS with master...
I20260620 11:37:09.883319 21579 heartbeater.cc:511] Master 127.2.243.190:33731 requested a full tablet report, sending...
I20260620 11:37:09.883646 21120 ts_manager.cc:194] Registered new tserver with Master: 3fc590aff8614380a34b2fb573a9f459 (127.2.243.131:37353)
I20260620 11:37:09.884047 21120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:37255
I20260620 11:37:09.890693 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:37:09.901207 21177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:09.893763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60130 (local address 127.2.243.190:33731)
0620 11:37:09.893820 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:09.893822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:09.893831 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:37:09.893854 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:09.893855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:09.893866 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:09.893924 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:09.894481 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.895245 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.896048 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.896342 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.896422 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:09.898425 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:09.898427 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:09.898427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:09.898435 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:09.899921 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:09.900343 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:09.900344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:09.900345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:09.900400 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:09.900662 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:09.900663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:09.900663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:09.900804 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:09.900927 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:09.901045 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 11:37:09.901108 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 11:37:09.902740 21120 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:37:09.904335 21120 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:37:09.910471 21587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:09.906380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:46695 (local address 127.0.0.1:38232)
0620 11:37:09.906505 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:09.906554 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:09.906618 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:09.907293 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:09.907293 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:09.907295 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:09.907330 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:09.907331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.908603 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.908604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:09.909495 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:09.909498 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.910192 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.910193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:09.910268 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:09.910281 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 11:37:09.910350 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 11:37:09.910402 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":77,"threads_started":1}
I20260620 11:37:09.910501 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:09.906511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38232 (local address 127.2.243.130:46695)
0620 11:37:09.907122 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:09.907127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:09.907136 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:37:09.907173 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:09.907175 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:09.907192 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:09.907396 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:09.907468 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.908498 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.909601 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.910069 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.910189 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:09.910249 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:09.910386 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 11:37:09.910423 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":82,"threads_started":1}
I20260620 11:37:09.910660 21590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:09.906401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:37353 (local address 127.0.0.1:50778)
0620 11:37:09.906693 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:09.906705 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:09.906756 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:09.907037 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:09.907041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:09.907043 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:09.907116 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:09.907119 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.908470 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.908471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:09.909519 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:09.909522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.910443 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.910444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:09.910508 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:09.910512 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:37:09.910543 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 11:37:09.910584 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":87,"threads_started":1}
I20260620 11:37:09.910846 21592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:09.906806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50068 (local address 127.2.243.129:33441)
0620 11:37:09.907017 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:09.907021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:09.907046 (+ 25us) server_negotiation.cc:407] Connection header received
0620 11:37:09.907085 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:09.907088 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:09.907102 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:09.907172 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:09.907385 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.908082 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.909617 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.910344 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.910413 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:09.910467 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:09.910709 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 11:37:09.910761 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":124,"threads_started":1}
I20260620 11:37:09.910868 21588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:09.906621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:33441 (local address 127.0.0.1:50068)
0620 11:37:09.906947 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:09.906957 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:09.907001 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:09.907251 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:09.907252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:09.907254 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:09.907304 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:09.907306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.908195 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.908196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:09.909530 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:09.909532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.910494 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.910495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:09.910564 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:09.910566 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:37:09.910610 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 11:37:09.910664 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":3,"thread_start_us":36,"threads_started":1}
I20260620 11:37:09.911214 21591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:09.906631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50778 (local address 127.2.243.131:37353)
0620 11:37:09.906786 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:09.906790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:09.906799 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:37:09.906835 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:09.906837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:09.906857 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:09.907021 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:09.907214 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.908361 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.909618 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.910340 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.910976 (+ 636us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:09.911047 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:09.911068 (+ 21us) server_negotiation.cc:299] Negotiation successful
0620 11:37:09.911139 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260620 11:37:09.911583 21379 tablet_service.cc:1467] Processing CreateTablet for tablet 68da49e58a074cfca21819845fa795ec (DEFAULT_TABLE table=test-table [id=26dffc95513d4a02b7eb10d933ed3bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:37:09.911932 21379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68da49e58a074cfca21819845fa795ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:09.911881 21513 tablet_service.cc:1467] Processing CreateTablet for tablet 68da49e58a074cfca21819845fa795ec (DEFAULT_TABLE table=test-table [id=26dffc95513d4a02b7eb10d933ed3bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:37:09.912168 21513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68da49e58a074cfca21819845fa795ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:09.914669 21593 tablet_bootstrap.cc:492] T 68da49e58a074cfca21819845fa795ec P a265b775d19944b78cbabd7ba25f8dcc: Bootstrap starting.
I20260620 11:37:09.914683 21594 tablet_bootstrap.cc:492] T 68da49e58a074cfca21819845fa795ec P 3fc590aff8614380a34b2fb573a9f459: Bootstrap starting.
I20260620 11:37:09.914987 21245 tablet_service.cc:1467] Processing CreateTablet for tablet 68da49e58a074cfca21819845fa795ec (DEFAULT_TABLE table=test-table [id=26dffc95513d4a02b7eb10d933ed3bec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:37:09.915318 21245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68da49e58a074cfca21819845fa795ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:09.915380 21593 tablet_bootstrap.cc:654] T 68da49e58a074cfca21819845fa795ec P a265b775d19944b78cbabd7ba25f8dcc: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:09.915387 21594 tablet_bootstrap.cc:654] T 68da49e58a074cfca21819845fa795ec P 3fc590aff8614380a34b2fb573a9f459: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:09.915861 21594 log.cc:824] T 68da49e58a074cfca21819845fa795ec P 3fc590aff8614380a34b2fb573a9f459: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:09.915915 21593 log.cc:824] T 68da49e58a074cfca21819845fa795ec P a265b775d19944b78cbabd7ba25f8dcc: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:09.916530 21594 tablet_bootstrap.cc:492] T 68da49e58a074cfca21819845fa795ec P 3fc590aff8614380a34b2fb573a9f459: No bootstrap required, opened a new log
I20260620 11:37:09.916630 21594 ts_tablet_manager.cc:1397] T 68da49e58a074cfca21819845fa795ec P 3fc590aff8614380a34b2fb573a9f459: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:37:09.916639 21593 tablet_bootstrap.cc:492] T 68da49e58a074cfca21819845fa795ec P a265b775d19944b78cbabd7ba25f8dcc: No bootstrap required, opened a new log
I20260620 11:37:09.916698 21593 ts_tablet_manager.cc:1397] T 68da49e58a074cfca21819845fa795ec P a265b775d19944b78cbabd7ba25f8dcc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:37:09.918108 21594 raft_consensus.cc:348] T 68da49e58a074cfca21819845fa795ec P 3fc590aff8614380a34b2fb573a9f459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a265b775d19944b78cbabd7ba25f8dcc" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 46695 } } peers { permanent_uuid: "3fc590aff8614380a34b2fb573a9f459" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 37353 } } peers { permanent_uuid: "b697a0ee76ff4b578ef0fff06518577b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 33441 } }
I20260620 11:37:09.918268 21594 raft_consensus.cc:374] T 68da49e58a074cfca21819845fa795ec P 3fc590aff8614380a34b2fb573a9f459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:09.918311 21594 raft_consensus.cc:729] T 68da49e58a074cfca21819845fa795ec P 3fc590aff8614380a34b2fb573a9f459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fc590aff8614380a34b2fb573a9f459, State: Initialized, Role: FOLLOWER
I20260620 11:37:09.918416 21594 consensus_queue.cc:260] T 68da49e58a074cfca21819845fa795ec P 3fc590aff8614380a34b2fb573a9f459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a265b775d19944b78cbabd7ba25f8dcc" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 46695 } } peers { permanent_uuid: "3fc590aff8614380a34b2fb573a9f459" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 37353 } } peers { permanent_uuid: "b697a0ee76ff4b578ef0fff06518577b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 33441 } }
I20260620 11:37:09.918697 21594 ts_tablet_manager.cc:1428] T 68da49e58a074cfca21819845fa795ec P 3fc590aff8614380a34b2fb573a9f459: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:37:09.918843 21579 heartbeater.cc:503] Master 127.2.243.190:33731 was elected leader, sending a full tablet report...
I20260620 11:37:09.918735 21593 raft_consensus.cc:348] T 68da49e58a074cfca21819845fa795ec P a265b775d19944b78cbabd7ba25f8dcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a265b775d19944b78cbabd7ba25f8dcc" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 46695 } } peers { permanent_uuid: "3fc590aff8614380a34b2fb573a9f459" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 37353 } } peers { permanent_uuid: "b697a0ee76ff4b578ef0fff06518577b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 33441 } }
I20260620 11:37:09.918926 21593 raft_consensus.cc:374] T 68da49e58a074cfca21819845fa795ec P a265b775d19944b78cbabd7ba25f8dcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:09.918898 21597 tablet_bootstrap.cc:492] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b: Bootstrap starting.
I20260620 11:37:09.918975 21593 raft_consensus.cc:729] T 68da49e58a074cfca21819845fa795ec P a265b775d19944b78cbabd7ba25f8dcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a265b775d19944b78cbabd7ba25f8dcc, State: Initialized, Role: FOLLOWER
I20260620 11:37:09.919114 21593 consensus_queue.cc:260] T 68da49e58a074cfca21819845fa795ec P a265b775d19944b78cbabd7ba25f8dcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a265b775d19944b78cbabd7ba25f8dcc" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 46695 } } peers { permanent_uuid: "3fc590aff8614380a34b2fb573a9f459" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 37353 } } peers { permanent_uuid: "b697a0ee76ff4b578ef0fff06518577b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 33441 } }
I20260620 11:37:09.919374 21593 ts_tablet_manager.cc:1428] T 68da49e58a074cfca21819845fa795ec P a265b775d19944b78cbabd7ba25f8dcc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:37:09.919488 21445 heartbeater.cc:503] Master 127.2.243.190:33731 was elected leader, sending a full tablet report...
I20260620 11:37:09.919617 21597 tablet_bootstrap.cc:654] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:09.920143 21597 log.cc:824] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:09.920794 21597 tablet_bootstrap.cc:492] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b: No bootstrap required, opened a new log
I20260620 11:37:09.920878 21597 ts_tablet_manager.cc:1397] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:37:09.922326 21597 raft_consensus.cc:348] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a265b775d19944b78cbabd7ba25f8dcc" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 46695 } } peers { permanent_uuid: "3fc590aff8614380a34b2fb573a9f459" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 37353 } } peers { permanent_uuid: "b697a0ee76ff4b578ef0fff06518577b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 33441 } }
I20260620 11:37:09.922492 21597 raft_consensus.cc:374] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:09.922536 21597 raft_consensus.cc:729] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b697a0ee76ff4b578ef0fff06518577b, State: Initialized, Role: FOLLOWER
I20260620 11:37:09.922643 21597 consensus_queue.cc:260] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a265b775d19944b78cbabd7ba25f8dcc" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 46695 } } peers { permanent_uuid: "3fc590aff8614380a34b2fb573a9f459" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 37353 } } peers { permanent_uuid: "b697a0ee76ff4b578ef0fff06518577b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 33441 } }
I20260620 11:37:09.922947 21597 ts_tablet_manager.cc:1428] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:37:09.922986 21311 heartbeater.cc:503] Master 127.2.243.190:33731 was elected leader, sending a full tablet report...
I20260620 11:37:09.942987 21601 raft_consensus.cc:482] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:37:09.943075 21601 raft_consensus.cc:504] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a265b775d19944b78cbabd7ba25f8dcc" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 46695 } } peers { permanent_uuid: "3fc590aff8614380a34b2fb573a9f459" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 37353 } } peers { permanent_uuid: "b697a0ee76ff4b578ef0fff06518577b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 33441 } }
I20260620 11:37:09.943468 21601 leader_election.cc:288] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a265b775d19944b78cbabd7ba25f8dcc (127.2.243.130:46695), 3fc590aff8614380a34b2fb573a9f459 (127.2.243.131:37353)
I20260620 11:37:09.947242 21591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:09.943672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:60537 (local address 127.2.243.131:37353)
0620 11:37:09.943723 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:09.943725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:09.943735 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:37:09.943814 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:09.943815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:09.943828 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:09.943924 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:09.944069 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.944833 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.946296 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.946885 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.946950 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:09.946958 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:09.947140 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 11:37:09.947177 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 11:37:09.947242 21589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:09.943732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:54929 (local address 127.2.243.130:46695)
0620 11:37:09.943774 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:09.943776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:09.943897 (+ 121us) server_negotiation.cc:407] Connection header received
0620 11:37:09.943928 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:09.943929 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:09.943938 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:09.944028 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:09.944158 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.944891 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.946300 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.946893 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.946950 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:09.946958 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:09.947140 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 11:37:09.947176 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:37:09.947237 21313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:09.943622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:37353 (local address 127.2.243.129:60537)
0620 11:37:09.943694 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:09.943706 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:09.943749 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:09.943937 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:09.943937 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:09.943939 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:09.944015 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:09.944016 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.944908 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.944908 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:09.946201 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:09.946204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.946965 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.946966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:09.947065 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:09.947068 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:37:09.947119 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 11:37:09.947150 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":49}
I20260620 11:37:09.947242 21602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:09.943629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:46695 (local address 127.2.243.129:54929)
0620 11:37:09.943798 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:09.943808 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:09.943861 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:09.944033 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:09.944034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:09.944035 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:09.944091 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:09.944092 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.944964 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.944965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:09.946201 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:09.946204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.946965 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.946966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:09.947042 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:09.947049 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 11:37:09.947107 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 11:37:09.947147 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":44,"thread_start_us":94,"threads_started":1}
I20260620 11:37:09.947623 21533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68da49e58a074cfca21819845fa795ec" candidate_uuid: "b697a0ee76ff4b578ef0fff06518577b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc590aff8614380a34b2fb573a9f459" is_pre_election: true
I20260620 11:37:09.947757 21533 raft_consensus.cc:2454] T 68da49e58a074cfca21819845fa795ec P 3fc590aff8614380a34b2fb573a9f459 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b697a0ee76ff4b578ef0fff06518577b in term 0.
I20260620 11:37:09.947990 21199 leader_election.cc:302] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fc590aff8614380a34b2fb573a9f459, b697a0ee76ff4b578ef0fff06518577b; no voters:
I20260620 11:37:09.948064 21399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68da49e58a074cfca21819845fa795ec" candidate_uuid: "b697a0ee76ff4b578ef0fff06518577b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a265b775d19944b78cbabd7ba25f8dcc" is_pre_election: true
I20260620 11:37:09.948112 21601 raft_consensus.cc:2781] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:37:09.948158 21601 raft_consensus.cc:482] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:37:09.948190 21601 raft_consensus.cc:3037] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:09.948204 21399 raft_consensus.cc:2454] T 68da49e58a074cfca21819845fa795ec P a265b775d19944b78cbabd7ba25f8dcc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b697a0ee76ff4b578ef0fff06518577b in term 0.
I20260620 11:37:09.948889 21601 raft_consensus.cc:504] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a265b775d19944b78cbabd7ba25f8dcc" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 46695 } } peers { permanent_uuid: "3fc590aff8614380a34b2fb573a9f459" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 37353 } } peers { permanent_uuid: "b697a0ee76ff4b578ef0fff06518577b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 33441 } }
I20260620 11:37:09.949038 21601 leader_election.cc:288] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b [CANDIDATE]: Term 1 election: Requested vote from peers a265b775d19944b78cbabd7ba25f8dcc (127.2.243.130:46695), 3fc590aff8614380a34b2fb573a9f459 (127.2.243.131:37353)
I20260620 11:37:09.949142 21399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68da49e58a074cfca21819845fa795ec" candidate_uuid: "b697a0ee76ff4b578ef0fff06518577b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a265b775d19944b78cbabd7ba25f8dcc"
I20260620 11:37:09.949185 21533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68da49e58a074cfca21819845fa795ec" candidate_uuid: "b697a0ee76ff4b578ef0fff06518577b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fc590aff8614380a34b2fb573a9f459"
I20260620 11:37:09.949250 21399 raft_consensus.cc:3037] T 68da49e58a074cfca21819845fa795ec P a265b775d19944b78cbabd7ba25f8dcc [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:09.949254 21533 raft_consensus.cc:3037] T 68da49e58a074cfca21819845fa795ec P 3fc590aff8614380a34b2fb573a9f459 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:09.949892 21399 raft_consensus.cc:2454] T 68da49e58a074cfca21819845fa795ec P a265b775d19944b78cbabd7ba25f8dcc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b697a0ee76ff4b578ef0fff06518577b in term 1.
I20260620 11:37:09.950009 21533 raft_consensus.cc:2454] T 68da49e58a074cfca21819845fa795ec P 3fc590aff8614380a34b2fb573a9f459 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b697a0ee76ff4b578ef0fff06518577b in term 1.
I20260620 11:37:09.950073 21200 leader_election.cc:302] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a265b775d19944b78cbabd7ba25f8dcc, b697a0ee76ff4b578ef0fff06518577b; no voters:
I20260620 11:37:09.950155 21601 raft_consensus.cc:2781] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:09.950274 21601 raft_consensus.cc:686] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b [term 1 LEADER]: Becoming Leader. State: Replica: b697a0ee76ff4b578ef0fff06518577b, State: Running, Role: LEADER
I20260620 11:37:09.950344 21601 consensus_queue.cc:237] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a265b775d19944b78cbabd7ba25f8dcc" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 46695 } } peers { permanent_uuid: "3fc590aff8614380a34b2fb573a9f459" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 37353 } } peers { permanent_uuid: "b697a0ee76ff4b578ef0fff06518577b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 33441 } }
I20260620 11:37:09.951133 21120 catalog_manager.cc:5495] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b reported cstate change: term changed from 0 to 1, leader changed from <none> to b697a0ee76ff4b578ef0fff06518577b (127.2.243.129). New cstate: current_term: 1 leader_uuid: "b697a0ee76ff4b578ef0fff06518577b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a265b775d19944b78cbabd7ba25f8dcc" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 46695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fc590aff8614380a34b2fb573a9f459" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 37353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b697a0ee76ff4b578ef0fff06518577b" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 33441 } health_report { overall_health: HEALTHY } } }
W20260620 11:37:09.956290 21446 tablet.cc:2369] T 68da49e58a074cfca21819845fa795ec P a265b775d19944b78cbabd7ba25f8dcc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:37:09.968097 21592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:09.965638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50074 (local address 127.2.243.129:33441)
0620 11:37:09.965683 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:09.965685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:09.965695 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:37:09.965764 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:09.965764 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:09.965788 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:09.965869 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:09.966002 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.966452 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.967307 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:09.967498 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:09.967582 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:09.967747 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:09.967838 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:09.967946 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:09.968012 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 11:37:09.968022 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:37:09.969738 21399 raft_consensus.cc:1261] T 68da49e58a074cfca21819845fa795ec P a265b775d19944b78cbabd7ba25f8dcc [term 1 FOLLOWER]: Refusing update from remote peer b697a0ee76ff4b578ef0fff06518577b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:37:09.969743 21533 raft_consensus.cc:1261] T 68da49e58a074cfca21819845fa795ec P 3fc590aff8614380a34b2fb573a9f459 [term 1 FOLLOWER]: Refusing update from remote peer b697a0ee76ff4b578ef0fff06518577b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:37:09.970237 21601 consensus_queue.cc:1035] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3fc590aff8614380a34b2fb573a9f459" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 37353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:37:09.970369 21603 consensus_queue.cc:1035] T 68da49e58a074cfca21819845fa795ec P b697a0ee76ff4b578ef0fff06518577b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a265b775d19944b78cbabd7ba25f8dcc" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 46695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:37:09.972931 21607 mvcc.cc:205] Tried to move back new op lower bound from 7298889441153748992 to 7298889441077366784. Current Snapshot: MvccSnapshot[applied={T|T < 7298889441153748992}]
I20260620 11:37:09.973038 21608 mvcc.cc:205] Tried to move back new op lower bound from 7298889441153748992 to 7298889441077366784. Current Snapshot: MvccSnapshot[applied={T|T < 7298889441153748992}]
I20260620 11:37:09.973212 21606 mvcc.cc:205] Tried to move back new op lower bound from 7298889441153748992 to 7298889441077366784. Current Snapshot: MvccSnapshot[applied={T|T < 7298889441153748992}]
I20260620 11:37:09.975236 21119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:37:09.975342 21119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:37:09.976122 21119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0fe2219c855b42e88ff1f3e7b48dee5e: Sending DeleteTablet for 3 replicas of tablet 68da49e58a074cfca21819845fa795ec
I20260620 11:37:09.976456 21513 tablet_service.cc:1514] Processing DeleteTablet for tablet 68da49e58a074cfca21819845fa795ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:37:09 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:50778
I20260620 11:37:09.976482 21379 tablet_service.cc:1514] Processing DeleteTablet for tablet 68da49e58a074cfca21819845fa795ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:37:09 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:38232
I20260620 11:37:09.976717 21617 tablet_replica.cc:330] T 68da49e58a074cfca21819845fa795ec P a265b775d19944b78cbabd7ba25f8dcc: stopping tablet replica
I20260620 11:37:09.976718 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 21182
I20260620 11:37:09.976809 21617 raft_consensus.cc:2229] T 68da49e58a074cfca21819845fa795ec P a265b775d19944b78cbabd7ba25f8dcc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:37:09.976898 21617 raft_consensus.cc:2258] T 68da49e58a074cfca21819845fa795ec P a265b775d19944b78cbabd7ba25f8dcc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:37:09.977054 21616 tablet_replica.cc:330] T 68da49e58a074cfca21819845fa795ec P 3fc590aff8614380a34b2fb573a9f459: stopping tablet replica
I20260620 11:37:09.977133 21616 raft_consensus.cc:2229] T 68da49e58a074cfca21819845fa795ec P 3fc590aff8614380a34b2fb573a9f459 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:37:09.977267 21616 raft_consensus.cc:2258] T 68da49e58a074cfca21819845fa795ec P 3fc590aff8614380a34b2fb573a9f459 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:37:09.977627 21616 ts_tablet_manager.cc:1905] T 68da49e58a074cfca21819845fa795ec P 3fc590aff8614380a34b2fb573a9f459: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:37:09.978075 21617 ts_tablet_manager.cc:1905] T 68da49e58a074cfca21819845fa795ec P a265b775d19944b78cbabd7ba25f8dcc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:37:09.979322 21616 ts_tablet_manager.cc:1918] T 68da49e58a074cfca21819845fa795ec P 3fc590aff8614380a34b2fb573a9f459: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:37:09.979390 21616 log.cc:1196] T 68da49e58a074cfca21819845fa795ec P 3fc590aff8614380a34b2fb573a9f459: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/68da49e58a074cfca21819845fa795ec
I20260620 11:37:09.979667 21616 ts_tablet_manager.cc:1939] T 68da49e58a074cfca21819845fa795ec P 3fc590aff8614380a34b2fb573a9f459: Deleting consensus metadata
I20260620 11:37:09.979707 21617 ts_tablet_manager.cc:1918] T 68da49e58a074cfca21819845fa795ec P a265b775d19944b78cbabd7ba25f8dcc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:37:09.979764 21617 log.cc:1196] T 68da49e58a074cfca21819845fa795ec P a265b775d19944b78cbabd7ba25f8dcc: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955192827595-3022-0/minicluster-data/ts-1/wal/wals/68da49e58a074cfca21819845fa795ec
I20260620 11:37:09.980013 21617 ts_tablet_manager.cc:1939] T 68da49e58a074cfca21819845fa795ec P a265b775d19944b78cbabd7ba25f8dcc: Deleting consensus metadata
I20260620 11:37:09.980129 21105 catalog_manager.cc:4841] TS 3fc590aff8614380a34b2fb573a9f459 (127.2.243.131:37353): tablet 68da49e58a074cfca21819845fa795ec (table test-table [id=26dffc95513d4a02b7eb10d933ed3bec]) successfully deleted
I20260620 11:37:09.980480 21106 catalog_manager.cc:4841] TS a265b775d19944b78cbabd7ba25f8dcc (127.2.243.130:46695): tablet 68da49e58a074cfca21819845fa795ec (table test-table [id=26dffc95513d4a02b7eb10d933ed3bec]) successfully deleted
W20260620 11:37:09.982388 21106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.243.129:33441 (error 108)
W20260620 11:37:09.983289 21588 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:09.982721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:33441 (local address 127.0.0.1:50076)
0620 11:37:09.982798 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:09.982809 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:09.982912 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:09.983257 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.129:33441: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 11:37:09.983454 21106 catalog_manager.cc:4584] TS b697a0ee76ff4b578ef0fff06518577b (127.2.243.129:33441): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 68da49e58a074cfca21819845fa795ec: Network error: Client connection negotiation failed: client connection to 127.2.243.129:33441: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 11:37:09.983497 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 21316
I20260620 11:37:09.989056 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 21450
I20260620 11:37:09.993789 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 21088
2026-06-20T11:37:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1891 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T11:37:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:10Z Disabled control of system clock
I20260620 11:37:10.021155 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:42387
--webserver_interface=127.2.243.190
--webserver_port=0
--builtin_ntp_servers=127.2.243.148:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:42387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 11:37:10.100217 21622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:10.100423 21622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:10.100443 21622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:10.101507 21622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:10.101543 21622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:10.101555 21622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:10.101568 21622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:10.101580 21622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:10.102800 21622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:42387
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:42387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:10.103000 21622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:10.103207 21622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:10.108047 21630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:10.108021 21628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:10.108072 21627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:10.108126 21622 server_base.cc:1034] running on GCE node
I20260620 11:37:10.108440 21622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:10.108712 21622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:10.109889 21622 hybrid_clock.cc:648] HybridClock initialized: now 1781955430109874 us; error 54 us; skew 500 ppm
I20260620 11:37:10.111196 21622 webserver.cc:460] Webserver started at http://127.2.243.190:39255/ using document root <none> and password file <none>
I20260620 11:37:10.111457 21622 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:10.111531 21622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:10.111671 21622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:10.112663 21622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "930d5cd1d9b74146a6225b3efe992cfe"
format_stamp: "Formatted at 2026-06-20 11:37:10 on dist-test-slave-ctbw"
server_key: "cb924dca0a276a0bda5adffc9267f33e"
server_key_iv: "67c368b1b78175681126da30d8d37305"
server_key_version: "encryptionkey@0"
I20260620 11:37:10.113039 21622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "930d5cd1d9b74146a6225b3efe992cfe"
format_stamp: "Formatted at 2026-06-20 11:37:10 on dist-test-slave-ctbw"
server_key: "cb924dca0a276a0bda5adffc9267f33e"
server_key_iv: "67c368b1b78175681126da30d8d37305"
server_key_version: "encryptionkey@0"
I20260620 11:37:10.114311 21622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:37:10.115092 21636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:10.115325 21622 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:37:10.115437 21622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "930d5cd1d9b74146a6225b3efe992cfe"
format_stamp: "Formatted at 2026-06-20 11:37:10 on dist-test-slave-ctbw"
server_key: "cb924dca0a276a0bda5adffc9267f33e"
server_key_iv: "67c368b1b78175681126da30d8d37305"
server_key_version: "encryptionkey@0"
I20260620 11:37:10.115515 21622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:10.136850 21622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:10.137197 21622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:10.137382 21622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:10.141727 21622 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:42387
I20260620 11:37:10.141793 21688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:42387 every 8 connection(s)
I20260620 11:37:10.142159 21622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:10.143317 21689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:10.145597 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 21622
I20260620 11:37:10.145733 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:10.145874 21689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe: Bootstrap starting.
I20260620 11:37:10.145925 3022 external_mini_cluster.cc:1428] Setting key e1b867e0200d4021f070f5d6b84dd914
I20260620 11:37:10.146600 21689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:10.146965 21689 log.cc:824] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:10.147683 21689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe: No bootstrap required, opened a new log
I20260620 11:37:10.149425 21689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "930d5cd1d9b74146a6225b3efe992cfe" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 42387 } }
I20260620 11:37:10.149587 21689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:10.149636 21689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 930d5cd1d9b74146a6225b3efe992cfe, State: Initialized, Role: FOLLOWER
I20260620 11:37:10.149765 21689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "930d5cd1d9b74146a6225b3efe992cfe" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 42387 } }
I20260620 11:37:10.149861 21689 raft_consensus.cc:388] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:10.149904 21689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:10.149950 21689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:10.150489 21689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "930d5cd1d9b74146a6225b3efe992cfe" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 42387 } }
I20260620 11:37:10.150614 21689 leader_election.cc:302] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 930d5cd1d9b74146a6225b3efe992cfe; no voters:
I20260620 11:37:10.150796 21689 leader_election.cc:288] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:10.150890 21694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:10.151037 21694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe [term 1 LEADER]: Becoming Leader. State: Replica: 930d5cd1d9b74146a6225b3efe992cfe, State: Running, Role: LEADER
I20260620 11:37:10.151098 21689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:10.151149 21694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "930d5cd1d9b74146a6225b3efe992cfe" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 42387 } }
I20260620 11:37:10.151479 21694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 930d5cd1d9b74146a6225b3efe992cfe. Latest consensus state: current_term: 1 leader_uuid: "930d5cd1d9b74146a6225b3efe992cfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "930d5cd1d9b74146a6225b3efe992cfe" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 42387 } } }
I20260620 11:37:10.151548 21694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:10.151760 21700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:10.152031 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:10.146476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34390 (local address 127.2.243.190:42387)
0620 11:37:10.146674 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:10.146683 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:10.146716 (+ 33us) server_negotiation.cc:407] Connection header received
0620 11:37:10.147254 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:10.147262 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:10.147321 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:10.147484 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:10.147689 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.148683 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.149479 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.150093 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.150239 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:10.151048 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:10.151052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:10.151052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:10.151255 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:10.151307 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:10.151324 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:10.151392 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:10.151749 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 11:37:10.151833 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":89,"threads_started":1}
I20260620 11:37:10.152513 21695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "930d5cd1d9b74146a6225b3efe992cfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "930d5cd1d9b74146a6225b3efe992cfe" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 42387 } } }
I20260620 11:37:10.152633 21695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:10.152659 21700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:10.153950 21700 catalog_manager.cc:1347] Generated new cluster ID: 87dea142e9164c87bf88a85fc7666886
I20260620 11:37:10.154008 21700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:37:10.170176 21700 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:10.170714 21700 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:10.182330 21700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe: Generated new TSK 0
I20260620 11:37:10.182531 21700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:37:10.189234 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:42387
--builtin_ntp_servers=127.2.243.148:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 11:37:10.272966 21713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:10.273151 21713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:10.273171 21713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:10.274278 21713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:10.274314 21713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:10.274338 21713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:37:10.275635 21713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:42387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:10.275897 21713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:10.276203 21713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:10.281091 21718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:10.281188 21719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:10.281231 21721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:10.281637 21713 server_base.cc:1034] running on GCE node
I20260620 11:37:10.281853 21713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:10.282112 21713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:10.283294 21713 hybrid_clock.cc:648] HybridClock initialized: now 1781955430283267 us; error 45 us; skew 500 ppm
I20260620 11:37:10.284571 21713 webserver.cc:460] Webserver started at http://127.2.243.129:40349/ using document root <none> and password file <none>
I20260620 11:37:10.284860 21713 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:10.284943 21713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:10.285090 21713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:10.286100 21713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "1e01c9d595e04100b7c7582a5bafd050"
format_stamp: "Formatted at 2026-06-20 11:37:10 on dist-test-slave-ctbw"
server_key: "7ec67992ca50b0a9c882effbfe1660cb"
server_key_iv: "678730f2a95c7de3383d8682da54a153"
server_key_version: "encryptionkey@0"
I20260620 11:37:10.286441 21713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "1e01c9d595e04100b7c7582a5bafd050"
format_stamp: "Formatted at 2026-06-20 11:37:10 on dist-test-slave-ctbw"
server_key: "7ec67992ca50b0a9c882effbfe1660cb"
server_key_iv: "678730f2a95c7de3383d8682da54a153"
server_key_version: "encryptionkey@0"
I20260620 11:37:10.287779 21713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:37:10.288501 21727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:10.288668 21713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:37:10.288766 21713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "1e01c9d595e04100b7c7582a5bafd050"
format_stamp: "Formatted at 2026-06-20 11:37:10 on dist-test-slave-ctbw"
server_key: "7ec67992ca50b0a9c882effbfe1660cb"
server_key_iv: "678730f2a95c7de3383d8682da54a153"
server_key_version: "encryptionkey@0"
I20260620 11:37:10.288847 21713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:10.296628 21713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:10.296902 21713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:10.297044 21713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:10.297251 21713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:37:10.297540 21713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:37:10.297596 21713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:10.297639 21713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:37:10.297665 21713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:10.303292 21713 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:44973
I20260620 11:37:10.303333 21840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:44973 every 8 connection(s)
I20260620 11:37:10.303640 21713 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:37:10.304179 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 21713
I20260620 11:37:10.304308 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:37:10.304518 3022 external_mini_cluster.cc:1428] Setting key 54ec53b8e07a9a83e2a8c5d1d43c4ae1
I20260620 11:37:10.305604 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:42387
--builtin_ntp_servers=127.2.243.148:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 11:37:10.309540 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:10.304344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:52939 (local address 127.2.243.190:42387)
0620 11:37:10.304392 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:10.304395 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:10.304673 (+ 278us) server_negotiation.cc:407] Connection header received
0620 11:37:10.305330 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:10.305330 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:10.305345 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:10.305398 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:10.306293 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.306824 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.307965 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.308277 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.308349 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:10.309188 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:10.309189 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:10.309190 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:10.309195 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:10.309212 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:10.309217 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:10.309274 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:10.309395 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 11:37:10.309428 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:37:10.309628 21843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:10.304309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:42387 (local address 127.2.243.129:52939)
0620 11:37:10.304518 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:10.304564 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:10.305206 (+ 642us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:10.305471 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:10.305483 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:10.305493 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:10.306200 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:10.306207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.307023 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.307024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:10.307882 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:10.307884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.308364 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.308365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:10.308467 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:10.309047 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:37:10.309047 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:10.309074 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:37:10.309095 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:10.309301 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:10.309310 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:10.309312 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:37:10.309367 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 11:37:10.309439 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":94,"threads_started":1}
I20260620 11:37:10.310112 21841 heartbeater.cc:347] Connected to a master server at 127.2.243.190:42387
I20260620 11:37:10.310199 21841 heartbeater.cc:464] Registering TS with master...
I20260620 11:37:10.310384 21841 heartbeater.cc:511] Master 127.2.243.190:42387 requested a full tablet report, sending...
I20260620 11:37:10.310957 21653 ts_manager.cc:194] Registered new tserver with Master: 1e01c9d595e04100b7c7582a5bafd050 (127.2.243.129:44973)
I20260620 11:37:10.311889 21653 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.243.129:52939
W20260620 11:37:10.386344 21844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:10.386540 21844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:10.386562 21844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:10.387661 21844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:10.387697 21844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:10.387722 21844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:37:10.389187 21844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:42387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:10.389381 21844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:10.389617 21844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:10.394618 21852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:10.394644 21849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:10.394726 21850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:10.394974 21844 server_base.cc:1034] running on GCE node
I20260620 11:37:10.395188 21844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:10.395447 21844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:10.396615 21844 hybrid_clock.cc:648] HybridClock initialized: now 1781955430396611 us; error 41 us; skew 500 ppm
I20260620 11:37:10.397979 21844 webserver.cc:460] Webserver started at http://127.2.243.130:42615/ using document root <none> and password file <none>
I20260620 11:37:10.398265 21844 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:10.398355 21844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:10.398511 21844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:10.399636 21844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "cacf6145221b4b5fbfb84c9ed9eb0975"
format_stamp: "Formatted at 2026-06-20 11:37:10 on dist-test-slave-ctbw"
server_key: "a47952f7357ab6e979b07eb133669b6c"
server_key_iv: "d3c11ce901deb90eb607a5137e09437d"
server_key_version: "encryptionkey@0"
I20260620 11:37:10.400017 21844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "cacf6145221b4b5fbfb84c9ed9eb0975"
format_stamp: "Formatted at 2026-06-20 11:37:10 on dist-test-slave-ctbw"
server_key: "a47952f7357ab6e979b07eb133669b6c"
server_key_iv: "d3c11ce901deb90eb607a5137e09437d"
server_key_version: "encryptionkey@0"
I20260620 11:37:10.401227 21844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 11:37:10.401921 21858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:10.402110 21844 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:37:10.402211 21844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "cacf6145221b4b5fbfb84c9ed9eb0975"
format_stamp: "Formatted at 2026-06-20 11:37:10 on dist-test-slave-ctbw"
server_key: "a47952f7357ab6e979b07eb133669b6c"
server_key_iv: "d3c11ce901deb90eb607a5137e09437d"
server_key_version: "encryptionkey@0"
I20260620 11:37:10.402294 21844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:10.411619 21844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:10.411906 21844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:10.412074 21844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:10.412274 21844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:37:10.412578 21844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:37:10.412637 21844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:10.412668 21844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:37:10.412700 21844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:10.418594 21844 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:39713
I20260620 11:37:10.418663 21971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:39713 every 8 connection(s)
I20260620 11:37:10.418998 21844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:37:10.420814 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 21844
I20260620 11:37:10.420928 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:37:10.421142 3022 external_mini_cluster.cc:1428] Setting key 8e5378dd1f509cc3539a549b194cb146
I20260620 11:37:10.422266 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:42387
--builtin_ntp_servers=127.2.243.148:42895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 11:37:10.424875 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:10.419777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:54175 (local address 127.2.243.190:42387)
0620 11:37:10.419864 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:10.419866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:10.420061 (+ 195us) server_negotiation.cc:407] Connection header received
0620 11:37:10.420527 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:10.420528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:10.420549 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:10.420600 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:10.421559 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.422150 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.423306 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.423623 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.423682 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:10.424618 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:10.424619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:10.424620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:10.424624 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:10.424638 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:10.424641 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:10.424685 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:10.424794 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 11:37:10.424819 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 11:37:10.425156 21974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:10.419725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:42387 (local address 127.2.243.130:54175)
0620 11:37:10.419924 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:10.419955 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:10.420432 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:10.420717 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:10.420722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:10.420732 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:10.421421 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:10.421426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.422242 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.422243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:10.423223 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:10.423225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.423717 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.423718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:10.423880 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:10.424447 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:37:10.424447 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:10.424485 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:37:10.424512 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:10.424708 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:10.424713 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:10.424716 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:37:10.424786 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 11:37:10.424895 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":101,"threads_started":1}
I20260620 11:37:10.425596 21972 heartbeater.cc:347] Connected to a master server at 127.2.243.190:42387
I20260620 11:37:10.425704 21972 heartbeater.cc:464] Registering TS with master...
I20260620 11:37:10.425891 21972 heartbeater.cc:511] Master 127.2.243.190:42387 requested a full tablet report, sending...
I20260620 11:37:10.426319 21653 ts_manager.cc:194] Registered new tserver with Master: cacf6145221b4b5fbfb84c9ed9eb0975 (127.2.243.130:39713)
I20260620 11:37:10.426852 21653 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.243.130:54175
W20260620 11:37:10.501729 21975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:10.501919 21975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:10.501940 21975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:10.503012 21975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:10.503044 21975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:10.503070 21975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:37:10.504374 21975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:42895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:42387
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:10.504627 21975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:10.504933 21975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:10.510105 21980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:10.510044 21981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:10.510123 21975 server_base.cc:1034] running on GCE node
W20260620 11:37:10.510054 21983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:10.510614 21975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:10.510926 21975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:10.512102 21975 hybrid_clock.cc:648] HybridClock initialized: now 1781955430512099 us; error 39 us; skew 500 ppm
I20260620 11:37:10.517184 21975 webserver.cc:460] Webserver started at http://127.2.243.131:34631/ using document root <none> and password file <none>
I20260620 11:37:10.517445 21975 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:10.517516 21975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:10.517683 21975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:10.518725 21975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "07359201d2be4d01af5f77f678099d94"
format_stamp: "Formatted at 2026-06-20 11:37:10 on dist-test-slave-ctbw"
server_key: "8c31b0e329c8de86caf4b570f5717db8"
server_key_iv: "de07d6a84ab7ab8bbfd339d9dbe8cbe2"
server_key_version: "encryptionkey@0"
I20260620 11:37:10.519109 21975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "07359201d2be4d01af5f77f678099d94"
format_stamp: "Formatted at 2026-06-20 11:37:10 on dist-test-slave-ctbw"
server_key: "8c31b0e329c8de86caf4b570f5717db8"
server_key_iv: "de07d6a84ab7ab8bbfd339d9dbe8cbe2"
server_key_version: "encryptionkey@0"
I20260620 11:37:10.520391 21975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:37:10.521134 21989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:10.521311 21975 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:37:10.521412 21975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "07359201d2be4d01af5f77f678099d94"
format_stamp: "Formatted at 2026-06-20 11:37:10 on dist-test-slave-ctbw"
server_key: "8c31b0e329c8de86caf4b570f5717db8"
server_key_iv: "de07d6a84ab7ab8bbfd339d9dbe8cbe2"
server_key_version: "encryptionkey@0"
I20260620 11:37:10.521504 21975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:10.534762 21975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:10.535094 21975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:10.535243 21975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:10.535461 21975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:37:10.535774 21975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:37:10.535843 21975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:10.535893 21975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:37:10.535914 21975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:10.541761 21975 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:35669
I20260620 11:37:10.541821 22102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:35669 every 8 connection(s)
I20260620 11:37:10.542130 21975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:37:10.547174 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:10.542844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:50203 (local address 127.2.243.190:42387)
0620 11:37:10.542897 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:10.542899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:10.543171 (+ 272us) server_negotiation.cc:407] Connection header received
0620 11:37:10.543630 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:10.543631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:10.543642 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:10.543690 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:10.544308 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.544806 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.545743 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.545970 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.546024 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:10.546851 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:10.546852 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:10.546852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:10.546857 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:10.546941 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:10.546946 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:10.546998 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:10.547096 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 11:37:10.547118 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 11:37:10.547403 22105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:10.542798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:42387 (local address 127.2.243.131:50203)
0620 11:37:10.543050 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:10.543082 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:10.543546 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:10.543751 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:10.543754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:10.543761 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:10.544241 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:10.544245 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.544889 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.544890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:10.545656 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:10.545658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.546052 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.546052 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:10.546154 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:10.546720 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:37:10.546721 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:10.546748 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:37:10.546770 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:10.547016 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:10.547021 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:10.547023 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:37:10.547080 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 11:37:10.547155 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":169,"threads_started":1}
I20260620 11:37:10.547710 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 21975
I20260620 11:37:10.547818 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:37:10.548034 3022 external_mini_cluster.cc:1428] Setting key a61b9ac903e2f4ace0de9f5adf5b5792
I20260620 11:37:10.548063 22103 heartbeater.cc:347] Connected to a master server at 127.2.243.190:42387
I20260620 11:37:10.548174 22103 heartbeater.cc:464] Registering TS with master...
I20260620 11:37:10.548375 22103 heartbeater.cc:511] Master 127.2.243.190:42387 requested a full tablet report, sending...
I20260620 11:37:10.548812 21652 ts_manager.cc:194] Registered new tserver with Master: 07359201d2be4d01af5f77f678099d94 (127.2.243.131:35669)
I20260620 11:37:10.549201 21652 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.243.131:50203
I20260620 11:37:10.559932 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:37:10.565270 21692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:10.561594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34402 (local address 127.2.243.190:42387)
0620 11:37:10.561678 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:10.561680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:10.561924 (+ 244us) server_negotiation.cc:407] Connection header received
0620 11:37:10.562049 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:10.562050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:10.562060 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:10.562129 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:10.562307 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.563066 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.564119 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.564301 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.564359 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:10.565051 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:10.565052 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:10.565052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:10.565057 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:10.565070 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:10.565074 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:10.565151 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:10.565207 (+ 56us) server_negotiation.cc:299] Negotiation successful
0620 11:37:10.565217 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:37:10.567186 21652 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:37:10.568764 21652 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:37:10.575151 22114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:10.571141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:39713 (local address 127.0.0.1:54786)
0620 11:37:10.571851 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:10.571862 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:10.571917 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:10.572143 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:10.572143 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:10.572145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:10.572174 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:10.572176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.573134 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.573135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:10.574138 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:10.574140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.574849 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.574850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:10.574940 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:10.574953 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 11:37:10.575034 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 11:37:10.575082 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"thread_start_us":67,"threads_started":1}
I20260620 11:37:10.575168 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:10.571324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54786 (local address 127.2.243.130:39713)
0620 11:37:10.571506 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:10.571511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:10.571992 (+ 481us) server_negotiation.cc:407] Connection header received
0620 11:37:10.572041 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:10.572043 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:10.572059 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:10.572135 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:10.572252 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.573043 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.574235 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.574730 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.574844 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:10.574952 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:10.575047 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 11:37:10.575082 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":103,"threads_started":1}
I20260620 11:37:10.575668 22112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:10.570929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:35669 (local address 127.0.0.1:33312)
0620 11:37:10.571283 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:10.571311 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:10.571374 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:10.571753 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:10.571756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:10.571759 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:10.571827 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:10.571828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.573118 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.573119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:10.574354 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:10.574357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.575421 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.575423 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:10.575482 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:10.575485 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:37:10.575552 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 11:37:10.575599 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":34,"spinlock_wait_cycles":384,"thread_start_us":88,"threads_started":1}
I20260620 11:37:10.575721 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:10.571099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33312 (local address 127.2.243.131:35669)
0620 11:37:10.571563 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:10.571567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:10.571577 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:37:10.571619 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:10.571620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:10.571633 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:10.571724 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:10.571916 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.573013 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.574455 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.575202 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.575279 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:10.575383 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:10.575609 (+ 226us) server_negotiation.cc:299] Negotiation successful
0620 11:37:10.575649 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":104,"threads_started":1}
I20260620 11:37:10.576004 21906 tablet_service.cc:1467] Processing CreateTablet for tablet b0b52a3b3f6f48d3b0774cf8bef74083 (DEFAULT_TABLE table=test-table [id=996020ab78b14f66bf86b5459cb5de4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:37:10.576402 21906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0b52a3b3f6f48d3b0774cf8bef74083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:10.576372 22037 tablet_service.cc:1467] Processing CreateTablet for tablet b0b52a3b3f6f48d3b0774cf8bef74083 (DEFAULT_TABLE table=test-table [id=996020ab78b14f66bf86b5459cb5de4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:37:10.576753 22037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0b52a3b3f6f48d3b0774cf8bef74083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:10.576756 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:10.570929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:44973 (local address 127.0.0.1:34842)
0620 11:37:10.571270 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:10.571311 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:10.571374 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:10.572438 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:10.572438 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:10.572440 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:10.572472 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:10.572474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.573757 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.573758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:10.574822 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:10.574824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.576433 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.576434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:10.576549 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:10.576554 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:37:10.576635 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 11:37:10.576674 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":64,"threads_started":1}
I20260620 11:37:10.577056 22115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:10.571073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34842 (local address 127.2.243.129:44973)
0620 11:37:10.572267 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:10.572272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:10.572282 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:37:10.572324 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:10.572326 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:10.572341 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:10.572424 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:10.572546 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.573649 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.574978 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.576243 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.576316 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:10.576395 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:10.576931 (+ 536us) server_negotiation.cc:299] Negotiation successful
0620 11:37:10.576973 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1151,"thread_start_us":146,"threads_started":1}
I20260620 11:37:10.578004 21775 tablet_service.cc:1467] Processing CreateTablet for tablet b0b52a3b3f6f48d3b0774cf8bef74083 (DEFAULT_TABLE table=test-table [id=996020ab78b14f66bf86b5459cb5de4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:37:10.578317 21775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0b52a3b3f6f48d3b0774cf8bef74083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:10.579540 22117 tablet_bootstrap.cc:492] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975: Bootstrap starting.
I20260620 11:37:10.580112 22118 tablet_bootstrap.cc:492] T b0b52a3b3f6f48d3b0774cf8bef74083 P 07359201d2be4d01af5f77f678099d94: Bootstrap starting.
I20260620 11:37:10.580289 22117 tablet_bootstrap.cc:654] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:10.580642 22117 log.cc:824] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:10.580740 22118 tablet_bootstrap.cc:654] T b0b52a3b3f6f48d3b0774cf8bef74083 P 07359201d2be4d01af5f77f678099d94: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:10.581056 22118 log.cc:824] T b0b52a3b3f6f48d3b0774cf8bef74083 P 07359201d2be4d01af5f77f678099d94: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:10.581426 22117 tablet_bootstrap.cc:492] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975: No bootstrap required, opened a new log
I20260620 11:37:10.581508 22117 ts_tablet_manager.cc:1397] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:37:10.581569 22121 tablet_bootstrap.cc:492] T b0b52a3b3f6f48d3b0774cf8bef74083 P 1e01c9d595e04100b7c7582a5bafd050: Bootstrap starting.
I20260620 11:37:10.581954 22118 tablet_bootstrap.cc:492] T b0b52a3b3f6f48d3b0774cf8bef74083 P 07359201d2be4d01af5f77f678099d94: No bootstrap required, opened a new log
I20260620 11:37:10.582031 22118 ts_tablet_manager.cc:1397] T b0b52a3b3f6f48d3b0774cf8bef74083 P 07359201d2be4d01af5f77f678099d94: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:37:10.582293 22121 tablet_bootstrap.cc:654] T b0b52a3b3f6f48d3b0774cf8bef74083 P 1e01c9d595e04100b7c7582a5bafd050: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:10.582660 22121 log.cc:824] T b0b52a3b3f6f48d3b0774cf8bef74083 P 1e01c9d595e04100b7c7582a5bafd050: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:10.583437 22121 tablet_bootstrap.cc:492] T b0b52a3b3f6f48d3b0774cf8bef74083 P 1e01c9d595e04100b7c7582a5bafd050: No bootstrap required, opened a new log
I20260620 11:37:10.583508 22121 ts_tablet_manager.cc:1397] T b0b52a3b3f6f48d3b0774cf8bef74083 P 1e01c9d595e04100b7c7582a5bafd050: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 11:37:10.584146 22117 raft_consensus.cc:348] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e01c9d595e04100b7c7582a5bafd050" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44973 } } peers { permanent_uuid: "07359201d2be4d01af5f77f678099d94" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 35669 } } peers { permanent_uuid: "cacf6145221b4b5fbfb84c9ed9eb0975" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39713 } }
I20260620 11:37:10.584298 22117 raft_consensus.cc:374] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:10.584343 22117 raft_consensus.cc:729] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cacf6145221b4b5fbfb84c9ed9eb0975, State: Initialized, Role: FOLLOWER
I20260620 11:37:10.584455 22117 consensus_queue.cc:260] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e01c9d595e04100b7c7582a5bafd050" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44973 } } peers { permanent_uuid: "07359201d2be4d01af5f77f678099d94" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 35669 } } peers { permanent_uuid: "cacf6145221b4b5fbfb84c9ed9eb0975" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39713 } }
I20260620 11:37:10.584774 22117 ts_tablet_manager.cc:1428] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:37:10.584805 21972 heartbeater.cc:503] Master 127.2.243.190:42387 was elected leader, sending a full tablet report...
I20260620 11:37:10.584784 22118 raft_consensus.cc:348] T b0b52a3b3f6f48d3b0774cf8bef74083 P 07359201d2be4d01af5f77f678099d94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e01c9d595e04100b7c7582a5bafd050" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44973 } } peers { permanent_uuid: "07359201d2be4d01af5f77f678099d94" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 35669 } } peers { permanent_uuid: "cacf6145221b4b5fbfb84c9ed9eb0975" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39713 } }
I20260620 11:37:10.584941 22118 raft_consensus.cc:374] T b0b52a3b3f6f48d3b0774cf8bef74083 P 07359201d2be4d01af5f77f678099d94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:10.584975 22118 raft_consensus.cc:729] T b0b52a3b3f6f48d3b0774cf8bef74083 P 07359201d2be4d01af5f77f678099d94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07359201d2be4d01af5f77f678099d94, State: Initialized, Role: FOLLOWER
I20260620 11:37:10.585088 22118 consensus_queue.cc:260] T b0b52a3b3f6f48d3b0774cf8bef74083 P 07359201d2be4d01af5f77f678099d94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e01c9d595e04100b7c7582a5bafd050" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44973 } } peers { permanent_uuid: "07359201d2be4d01af5f77f678099d94" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 35669 } } peers { permanent_uuid: "cacf6145221b4b5fbfb84c9ed9eb0975" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39713 } }
I20260620 11:37:10.585333 22118 ts_tablet_manager.cc:1428] T b0b52a3b3f6f48d3b0774cf8bef74083 P 07359201d2be4d01af5f77f678099d94: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 11:37:10.585381 22103 heartbeater.cc:503] Master 127.2.243.190:42387 was elected leader, sending a full tablet report...
I20260620 11:37:10.586233 22121 raft_consensus.cc:348] T b0b52a3b3f6f48d3b0774cf8bef74083 P 1e01c9d595e04100b7c7582a5bafd050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e01c9d595e04100b7c7582a5bafd050" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44973 } } peers { permanent_uuid: "07359201d2be4d01af5f77f678099d94" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 35669 } } peers { permanent_uuid: "cacf6145221b4b5fbfb84c9ed9eb0975" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39713 } }
I20260620 11:37:10.586346 22121 raft_consensus.cc:374] T b0b52a3b3f6f48d3b0774cf8bef74083 P 1e01c9d595e04100b7c7582a5bafd050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:10.586380 22121 raft_consensus.cc:729] T b0b52a3b3f6f48d3b0774cf8bef74083 P 1e01c9d595e04100b7c7582a5bafd050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e01c9d595e04100b7c7582a5bafd050, State: Initialized, Role: FOLLOWER
I20260620 11:37:10.586513 22121 consensus_queue.cc:260] T b0b52a3b3f6f48d3b0774cf8bef74083 P 1e01c9d595e04100b7c7582a5bafd050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e01c9d595e04100b7c7582a5bafd050" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44973 } } peers { permanent_uuid: "07359201d2be4d01af5f77f678099d94" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 35669 } } peers { permanent_uuid: "cacf6145221b4b5fbfb84c9ed9eb0975" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39713 } }
I20260620 11:37:10.586761 22121 ts_tablet_manager.cc:1428] T b0b52a3b3f6f48d3b0774cf8bef74083 P 1e01c9d595e04100b7c7582a5bafd050: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 11:37:10.586846 21841 heartbeater.cc:503] Master 127.2.243.190:42387 was elected leader, sending a full tablet report...
W20260620 11:37:10.669574 21973 tablet.cc:2369] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:37:10.763257 22123 raft_consensus.cc:482] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:37:10.763396 22123 raft_consensus.cc:504] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e01c9d595e04100b7c7582a5bafd050" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44973 } } peers { permanent_uuid: "07359201d2be4d01af5f77f678099d94" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 35669 } } peers { permanent_uuid: "cacf6145221b4b5fbfb84c9ed9eb0975" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39713 } }
I20260620 11:37:10.763815 22123 leader_election.cc:288] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e01c9d595e04100b7c7582a5bafd050 (127.2.243.129:44973), 07359201d2be4d01af5f77f678099d94 (127.2.243.131:35669)
I20260620 11:37:10.766767 22113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:10.764123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:48321 (local address 127.2.243.131:35669)
0620 11:37:10.764166 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:10.764168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:10.764272 (+ 104us) server_negotiation.cc:407] Connection header received
0620 11:37:10.764304 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:10.764305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:10.764314 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:10.764383 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:10.764559 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.765016 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.765979 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.766316 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.766507 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:10.766515 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:10.766657 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 11:37:10.766697 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:37:10.766775 21974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:10.763987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:44973 (local address 127.2.243.130:40417)
0620 11:37:10.764044 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:10.764056 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:10.764101 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:10.764292 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:10.764293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:10.764295 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:10.764370 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:10.764372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.765089 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.765091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:10.765831 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:10.765833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.766497 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.766498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:10.766569 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:10.766574 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:37:10.766657 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 11:37:10.766697 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":1}
I20260620 11:37:10.766782 22115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:10.764070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:40417 (local address 127.2.243.129:44973)
0620 11:37:10.764111 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:10.764113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:10.764142 (+ 29us) server_negotiation.cc:407] Connection header received
0620 11:37:10.764171 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:10.764173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:10.764184 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:10.764248 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:10.764455 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.764991 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.765974 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.766313 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.766368 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:10.766374 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:10.766681 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 11:37:10.766720 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:37:10.766783 22126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:10.764011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:35669 (local address 127.2.243.130:48321)
0620 11:37:10.764195 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:10.764204 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:10.764253 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:10.764390 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:10.764391 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:10.764392 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:10.764484 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:10.764485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.765127 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.765128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:10.765844 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:10.765846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.766514 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.766515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:10.766571 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:10.766573 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:37:10.766618 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 11:37:10.766683 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":103,"threads_started":1}
I20260620 11:37:10.767153 21795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0b52a3b3f6f48d3b0774cf8bef74083" candidate_uuid: "cacf6145221b4b5fbfb84c9ed9eb0975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e01c9d595e04100b7c7582a5bafd050" is_pre_election: true
I20260620 11:37:10.767171 22057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0b52a3b3f6f48d3b0774cf8bef74083" candidate_uuid: "cacf6145221b4b5fbfb84c9ed9eb0975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07359201d2be4d01af5f77f678099d94" is_pre_election: true
I20260620 11:37:10.767316 21795 raft_consensus.cc:2454] T b0b52a3b3f6f48d3b0774cf8bef74083 P 1e01c9d595e04100b7c7582a5bafd050 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cacf6145221b4b5fbfb84c9ed9eb0975 in term 0.
I20260620 11:37:10.767336 22057 raft_consensus.cc:2454] T b0b52a3b3f6f48d3b0774cf8bef74083 P 07359201d2be4d01af5f77f678099d94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cacf6145221b4b5fbfb84c9ed9eb0975 in term 0.
I20260620 11:37:10.767565 21860 leader_election.cc:302] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07359201d2be4d01af5f77f678099d94, cacf6145221b4b5fbfb84c9ed9eb0975; no voters:
I20260620 11:37:10.767732 22123 raft_consensus.cc:2781] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:37:10.767812 22123 raft_consensus.cc:482] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:37:10.767844 22123 raft_consensus.cc:3037] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:10.768527 22123 raft_consensus.cc:504] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e01c9d595e04100b7c7582a5bafd050" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44973 } } peers { permanent_uuid: "07359201d2be4d01af5f77f678099d94" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 35669 } } peers { permanent_uuid: "cacf6145221b4b5fbfb84c9ed9eb0975" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39713 } }
I20260620 11:37:10.768678 22123 leader_election.cc:288] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975 [CANDIDATE]: Term 1 election: Requested vote from peers 1e01c9d595e04100b7c7582a5bafd050 (127.2.243.129:44973), 07359201d2be4d01af5f77f678099d94 (127.2.243.131:35669)
I20260620 11:37:10.768836 22057 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0b52a3b3f6f48d3b0774cf8bef74083" candidate_uuid: "cacf6145221b4b5fbfb84c9ed9eb0975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07359201d2be4d01af5f77f678099d94"
I20260620 11:37:10.768935 22057 raft_consensus.cc:3037] T b0b52a3b3f6f48d3b0774cf8bef74083 P 07359201d2be4d01af5f77f678099d94 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:10.769058 21795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0b52a3b3f6f48d3b0774cf8bef74083" candidate_uuid: "cacf6145221b4b5fbfb84c9ed9eb0975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e01c9d595e04100b7c7582a5bafd050"
I20260620 11:37:10.769158 21795 raft_consensus.cc:3037] T b0b52a3b3f6f48d3b0774cf8bef74083 P 1e01c9d595e04100b7c7582a5bafd050 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:10.769706 22057 raft_consensus.cc:2454] T b0b52a3b3f6f48d3b0774cf8bef74083 P 07359201d2be4d01af5f77f678099d94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cacf6145221b4b5fbfb84c9ed9eb0975 in term 1.
I20260620 11:37:10.769886 21860 leader_election.cc:302] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07359201d2be4d01af5f77f678099d94, cacf6145221b4b5fbfb84c9ed9eb0975; no voters:
I20260620 11:37:10.769996 22123 raft_consensus.cc:2781] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:10.769994 21795 raft_consensus.cc:2454] T b0b52a3b3f6f48d3b0774cf8bef74083 P 1e01c9d595e04100b7c7582a5bafd050 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cacf6145221b4b5fbfb84c9ed9eb0975 in term 1.
I20260620 11:37:10.770164 22123 raft_consensus.cc:686] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975 [term 1 LEADER]: Becoming Leader. State: Replica: cacf6145221b4b5fbfb84c9ed9eb0975, State: Running, Role: LEADER
I20260620 11:37:10.770263 22123 consensus_queue.cc:237] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e01c9d595e04100b7c7582a5bafd050" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44973 } } peers { permanent_uuid: "07359201d2be4d01af5f77f678099d94" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 35669 } } peers { permanent_uuid: "cacf6145221b4b5fbfb84c9ed9eb0975" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39713 } }
I20260620 11:37:10.771173 21652 catalog_manager.cc:5495] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975 reported cstate change: term changed from 0 to 1, leader changed from <none> to cacf6145221b4b5fbfb84c9ed9eb0975 (127.2.243.130). New cstate: current_term: 1 leader_uuid: "cacf6145221b4b5fbfb84c9ed9eb0975" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e01c9d595e04100b7c7582a5bafd050" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07359201d2be4d01af5f77f678099d94" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 35669 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cacf6145221b4b5fbfb84c9ed9eb0975" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 39713 } health_report { overall_health: HEALTHY } } }
W20260620 11:37:10.792644 22104 tablet.cc:2369] T b0b52a3b3f6f48d3b0774cf8bef74083 P 07359201d2be4d01af5f77f678099d94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:37:10.800978 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:10.798537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54788 (local address 127.2.243.130:39713)
0620 11:37:10.798592 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:10.798594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:10.798602 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:37:10.798660 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:10.798661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:10.798679 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:10.798740 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:10.799052 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.799498 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.800218 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:10.800395 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:10.800447 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:10.800614 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:10.800700 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:10.800771 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:10.800818 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 11:37:10.800826 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 11:37:10.802822 21795 raft_consensus.cc:1261] T b0b52a3b3f6f48d3b0774cf8bef74083 P 1e01c9d595e04100b7c7582a5bafd050 [term 1 FOLLOWER]: Refusing update from remote peer cacf6145221b4b5fbfb84c9ed9eb0975: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:37:10.802829 22057 raft_consensus.cc:1261] T b0b52a3b3f6f48d3b0774cf8bef74083 P 07359201d2be4d01af5f77f678099d94 [term 1 FOLLOWER]: Refusing update from remote peer cacf6145221b4b5fbfb84c9ed9eb0975: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:37:10.803305 22123 consensus_queue.cc:1035] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07359201d2be4d01af5f77f678099d94" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 35669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:37:10.803465 22127 consensus_queue.cc:1035] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e01c9d595e04100b7c7582a5bafd050" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 11:37:10.804237 21842 tablet.cc:2369] T b0b52a3b3f6f48d3b0774cf8bef74083 P 1e01c9d595e04100b7c7582a5bafd050: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:37:10.805789 22131 mvcc.cc:205] Tried to move back new op lower bound from 7298889444565671936 to 7298889444435746816. Current Snapshot: MvccSnapshot[applied={T|T < 7298889444565671936}]
I20260620 11:37:10.806340 22132 mvcc.cc:205] Tried to move back new op lower bound from 7298889444565671936 to 7298889444435746816. Current Snapshot: MvccSnapshot[applied={T|T < 7298889444565671936}]
I20260620 11:37:10.808584 21653 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:37:10.808686 21653 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:37:10.809435 21653 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 930d5cd1d9b74146a6225b3efe992cfe: Sending DeleteTablet for 3 replicas of tablet b0b52a3b3f6f48d3b0774cf8bef74083
I20260620 11:37:10.809787 22037 tablet_service.cc:1514] Processing DeleteTablet for tablet b0b52a3b3f6f48d3b0774cf8bef74083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:37:10 UTC) from {username='slave'} at 127.0.0.1:33312
I20260620 11:37:10.809840 21906 tablet_service.cc:1514] Processing DeleteTablet for tablet b0b52a3b3f6f48d3b0774cf8bef74083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:37:10 UTC) from {username='slave'} at 127.0.0.1:54786
I20260620 11:37:10.809859 21775 tablet_service.cc:1514] Processing DeleteTablet for tablet b0b52a3b3f6f48d3b0774cf8bef74083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:37:10 UTC) from {username='slave'} at 127.0.0.1:34842
I20260620 11:37:10.810034 22141 tablet_replica.cc:330] T b0b52a3b3f6f48d3b0774cf8bef74083 P 1e01c9d595e04100b7c7582a5bafd050: stopping tablet replica
I20260620 11:37:10.810129 22141 raft_consensus.cc:2229] T b0b52a3b3f6f48d3b0774cf8bef74083 P 1e01c9d595e04100b7c7582a5bafd050 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:37:10.810230 22141 raft_consensus.cc:2258] T b0b52a3b3f6f48d3b0774cf8bef74083 P 1e01c9d595e04100b7c7582a5bafd050 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:37:10.810375 22142 tablet_replica.cc:330] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975: stopping tablet replica
I20260620 11:37:10.810377 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 21713
I20260620 11:37:10.810465 22142 raft_consensus.cc:2229] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:37:10.810580 22142 raft_consensus.cc:2258] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:37:10.812341 22142 ts_tablet_manager.cc:1905] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:37:10.814289 22142 ts_tablet_manager.cc:1918] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:37:10.814368 22142 log.cc:1196] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-1/wal/wals/b0b52a3b3f6f48d3b0774cf8bef74083
I20260620 11:37:10.814603 22142 ts_tablet_manager.cc:1939] T b0b52a3b3f6f48d3b0774cf8bef74083 P cacf6145221b4b5fbfb84c9ed9eb0975: Deleting consensus metadata
I20260620 11:37:10.815289 22140 tablet_replica.cc:330] T b0b52a3b3f6f48d3b0774cf8bef74083 P 07359201d2be4d01af5f77f678099d94: stopping tablet replica
I20260620 11:37:10.815394 22140 raft_consensus.cc:2229] T b0b52a3b3f6f48d3b0774cf8bef74083 P 07359201d2be4d01af5f77f678099d94 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:37:10.815500 22140 raft_consensus.cc:2258] T b0b52a3b3f6f48d3b0774cf8bef74083 P 07359201d2be4d01af5f77f678099d94 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:37:10.815953 22140 ts_tablet_manager.cc:1905] T b0b52a3b3f6f48d3b0774cf8bef74083 P 07359201d2be4d01af5f77f678099d94: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:37:10.816972 21639 catalog_manager.cc:4841] TS cacf6145221b4b5fbfb84c9ed9eb0975 (127.2.243.130:39713): tablet b0b52a3b3f6f48d3b0774cf8bef74083 (table test-table [id=996020ab78b14f66bf86b5459cb5de4f]) successfully deleted
W20260620 11:37:10.817107 21639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.243.129:44973 (error 108)
I20260620 11:37:10.817287 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 21844
I20260620 11:37:10.817420 22140 ts_tablet_manager.cc:1918] T b0b52a3b3f6f48d3b0774cf8bef74083 P 07359201d2be4d01af5f77f678099d94: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:37:10.817492 22140 log.cc:1196] T b0b52a3b3f6f48d3b0774cf8bef74083 P 07359201d2be4d01af5f77f678099d94: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/b0b52a3b3f6f48d3b0774cf8bef74083
I20260620 11:37:10.817734 22140 ts_tablet_manager.cc:1939] T b0b52a3b3f6f48d3b0774cf8bef74083 P 07359201d2be4d01af5f77f678099d94: Deleting consensus metadata
I20260620 11:37:10.818147 21638 catalog_manager.cc:4841] TS 07359201d2be4d01af5f77f678099d94 (127.2.243.131:35669): tablet b0b52a3b3f6f48d3b0774cf8bef74083 (table test-table [id=996020ab78b14f66bf86b5459cb5de4f]) successfully deleted
I20260620 11:37:10.822583 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 21975
I20260620 11:37:10.822731 22111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:10.822587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:44973 (local address 127.0.0.1:34854)
0620 11:37:10.822650 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:10.822693 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.129:44973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 11:37:10.823338 21639 catalog_manager.cc:4584] TS 1e01c9d595e04100b7c7582a5bafd050 (127.2.243.129:44973): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b0b52a3b3f6f48d3b0774cf8bef74083: Network error: Client connection negotiation failed: client connection to 127.2.243.129:44973: connect: Connection refused (error 111)
I20260620 11:37:10.827198 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 21622
2026-06-20T11:37:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (832 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:10 dist-test-slave-ctbw krb5kdc[22146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:10 dist-test-slave-ctbw krb5kdc[22146](info): set up 2 sockets
Jun 20 11:37:10 dist-test-slave-ctbw krb5kdc[22146](info): commencing operation
krb5kdc: starting...
W20260620 11:37:11.874950 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:11 dist-test-slave-ctbw krb5kdc[22146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:37:11.993562 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:44123
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:43285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:44123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:12.073359 22159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:12.073632 22159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:12.073676 22159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:12.074914 22159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:12.074997 22159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:12.075021 22159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:12.075043 22159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:12.075069 22159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:12.076577 22159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:43285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:44123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:44123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:12.076875 22159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:12.077119 22159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:12.082229 22164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:12.082199 22167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:12.082336 22159 server_base.cc:1034] running on GCE node
W20260620 11:37:12.082243 22165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:12.083009 22159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:12.083289 22159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:12.084431 22159 hybrid_clock.cc:648] HybridClock initialized: now 1781955432084422 us; error 31 us; skew 500 ppm
Jun 20 11:37:12 dist-test-slave-ctbw krb5kdc[22146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955432, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:12.090063 22159 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:37:12.090523 22159 webserver.cc:460] Webserver started at http://127.2.243.190:38421/ using document root <none> and password file <none>
I20260620 11:37:12.090777 22159 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:12.090826 22159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:12.091014 22159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:12.092108 22159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "f104c2b8bb764e7d9f787ecff741fc6d"
format_stamp: "Formatted at 2026-06-20 11:37:12 on dist-test-slave-ctbw"
server_key: "37d0e02ed63d13e03bf4e9c1e2ea5704"
server_key_iv: "49d937f829871a4cac445c552fe3d8dc"
server_key_version: "encryptionkey@0"
I20260620 11:37:12.092490 22159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "f104c2b8bb764e7d9f787ecff741fc6d"
format_stamp: "Formatted at 2026-06-20 11:37:12 on dist-test-slave-ctbw"
server_key: "37d0e02ed63d13e03bf4e9c1e2ea5704"
server_key_iv: "49d937f829871a4cac445c552fe3d8dc"
server_key_version: "encryptionkey@0"
I20260620 11:37:12.093799 22159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:37:12.094544 22174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:12.094729 22159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:37:12.094826 22159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "f104c2b8bb764e7d9f787ecff741fc6d"
format_stamp: "Formatted at 2026-06-20 11:37:12 on dist-test-slave-ctbw"
server_key: "37d0e02ed63d13e03bf4e9c1e2ea5704"
server_key_iv: "49d937f829871a4cac445c552fe3d8dc"
server_key_version: "encryptionkey@0"
I20260620 11:37:12.094967 22159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:12.116209 22159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:12.118148 22159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:12.118335 22159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:12.122364 22159 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:44123
I20260620 11:37:12.122398 22226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:44123 every 8 connection(s)
I20260620 11:37:12.122754 22159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:12.123687 22227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:12.125934 22227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d: Bootstrap starting.
I20260620 11:37:12.126449 22227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:12.126677 22227 log.cc:824] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:12.127404 22227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d: No bootstrap required, opened a new log
I20260620 11:37:12.128701 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 22159
I20260620 11:37:12.128831 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:12.129017 3022 external_mini_cluster.cc:1428] Setting key 1dfaca04fc1739ca11dec3ebc8c07d2e
I20260620 11:37:12.129359 22227 raft_consensus.cc:348] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f104c2b8bb764e7d9f787ecff741fc6d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 44123 } }
I20260620 11:37:12.129539 22227 raft_consensus.cc:374] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:12.129630 22227 raft_consensus.cc:729] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f104c2b8bb764e7d9f787ecff741fc6d, State: Initialized, Role: FOLLOWER
I20260620 11:37:12.129802 22227 consensus_queue.cc:260] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f104c2b8bb764e7d9f787ecff741fc6d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 44123 } }
I20260620 11:37:12.129910 22227 raft_consensus.cc:388] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:12.129936 22227 raft_consensus.cc:482] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:12.129997 22227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:12.131157 22227 raft_consensus.cc:504] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f104c2b8bb764e7d9f787ecff741fc6d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 44123 } }
I20260620 11:37:12.131248 22227 leader_election.cc:302] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f104c2b8bb764e7d9f787ecff741fc6d; no voters:
I20260620 11:37:12.131453 22227 leader_election.cc:288] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:12.131482 22232 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:12.131593 22232 raft_consensus.cc:686] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d [term 1 LEADER]: Becoming Leader. State: Replica: f104c2b8bb764e7d9f787ecff741fc6d, State: Running, Role: LEADER
I20260620 11:37:12.131736 22227 sys_catalog.cc:564] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:12.131768 22232 consensus_queue.cc:237] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f104c2b8bb764e7d9f787ecff741fc6d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 44123 } }
I20260620 11:37:12.132102 22232 sys_catalog.cc:455] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f104c2b8bb764e7d9f787ecff741fc6d. Latest consensus state: current_term: 1 leader_uuid: "f104c2b8bb764e7d9f787ecff741fc6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f104c2b8bb764e7d9f787ecff741fc6d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 44123 } } }
I20260620 11:37:12.132113 22234 sys_catalog.cc:455] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f104c2b8bb764e7d9f787ecff741fc6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f104c2b8bb764e7d9f787ecff741fc6d" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 44123 } } }
I20260620 11:37:12.132195 22232 sys_catalog.cc:458] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:12.132211 22234 sys_catalog.cc:458] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:12.132442 22238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:12.132975 22238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:12.134582 22238 catalog_manager.cc:1347] Generated new cluster ID: 31b41d96fde64a46b3aafb91f594072b
I20260620 11:37:12.134959 22238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:37:12 dist-test-slave-ctbw krb5kdc[22146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:37:12.140491 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:12.129554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41102 (local address 127.2.243.190:44123)
0620 11:37:12.129761 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:12.129771 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:12.129803 (+ 32us) server_negotiation.cc:407] Connection header received
0620 11:37:12.129948 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:12.129955 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:12.130012 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:12.130161 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:12.131002 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.132507 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.133546 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.134032 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.134144 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:12.136641 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:12.136645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:12.136646 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:12.136658 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:12.138488 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:12.138907 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:12.138911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:12.138911 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:12.139024 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:12.139252 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:12.139253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:12.139253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:12.139509 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:12.140166 (+ 657us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:12.140226 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 11:37:12.140334 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":86,"threads_started":1}
I20260620 11:37:12.149554 22238 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:12.150307 22238 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:12.159142 22238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d: Generated new TSK 0
I20260620 11:37:12.159394 22238 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.2.243.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
Entry for principal kudu/127.2.243.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab.
I20260620 11:37:12.190092 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:44123
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:43285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:12.270565 22253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:12.270813 22253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:12.270852 22253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:12.272004 22253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:12.272079 22253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:12.272125 22253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:37:12.273590 22253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:43285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.129.keytab
--principal=kudu/127.2.243.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:12.273882 22253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:12.274121 22253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:12.279132 22258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:12.279132 22259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:12.279202 22261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:12.279431 22253 server_base.cc:1034] running on GCE node
I20260620 11:37:12.279647 22253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:12.279896 22253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:12.281072 22253 hybrid_clock.cc:648] HybridClock initialized: now 1781955432281057 us; error 48 us; skew 500 ppm
Jun 20 11:37:12 dist-test-slave-ctbw krb5kdc[22146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955432, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:12.283699 22253 init.cc:377] Logged in from keytab as kudu/127.2.243.129@KRBTEST.COM (short username kudu)
I20260620 11:37:12.284160 22253 webserver.cc:460] Webserver started at http://127.2.243.129:34485/ using document root <none> and password file <none>
I20260620 11:37:12.284350 22253 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:12.284391 22253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:12.284499 22253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:12.285539 22253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "e6cc25cb87d946c0a1c91fa43e7dad59"
format_stamp: "Formatted at 2026-06-20 11:37:12 on dist-test-slave-ctbw"
server_key: "e9f034fcf35a8b72ed61fd1c69cd8b1d"
server_key_iv: "baa0e7a6ecb4b8b4cad342dfff2f1ffc"
server_key_version: "encryptionkey@0"
I20260620 11:37:12.285894 22253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "e6cc25cb87d946c0a1c91fa43e7dad59"
format_stamp: "Formatted at 2026-06-20 11:37:12 on dist-test-slave-ctbw"
server_key: "e9f034fcf35a8b72ed61fd1c69cd8b1d"
server_key_iv: "baa0e7a6ecb4b8b4cad342dfff2f1ffc"
server_key_version: "encryptionkey@0"
I20260620 11:37:12.287261 22253 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:37:12.288026 22268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:12.288200 22253 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:37:12.288282 22253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "e6cc25cb87d946c0a1c91fa43e7dad59"
format_stamp: "Formatted at 2026-06-20 11:37:12 on dist-test-slave-ctbw"
server_key: "e9f034fcf35a8b72ed61fd1c69cd8b1d"
server_key_iv: "baa0e7a6ecb4b8b4cad342dfff2f1ffc"
server_key_version: "encryptionkey@0"
I20260620 11:37:12.288338 22253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:12.296914 22253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:12.299309 22253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:12.299417 22253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:12.299578 22253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:37:12.299811 22253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:37:12.299856 22253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:12.299878 22253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:37:12.299890 22253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:12.305543 22253 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:35931
I20260620 11:37:12.305634 22381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:35931 every 8 connection(s)
I20260620 11:37:12.305964 22253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
Jun 20 11:37:12 dist-test-slave-ctbw krb5kdc[22146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955432, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.129@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:37:12.315034 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:12.306730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:44649 (local address 127.2.243.190:44123)
0620 11:37:12.306798 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:12.306799 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:12.307034 (+ 235us) server_negotiation.cc:407] Connection header received
0620 11:37:12.307499 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:12.307500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:12.307513 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:12.307574 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:12.308499 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.309007 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.309943 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.310137 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.310188 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:12.312341 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:12.312342 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:12.312343 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:12.312349 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:12.313788 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:12.314156 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:12.314157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:12.314158 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:12.314205 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:12.314413 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:12.314413 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:12.314413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:12.314510 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:12.314559 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:12.314787 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 11:37:12.314907 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 11:37:12.315102 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 22253
I20260620 11:37:12.315212 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:37:12.315223 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:12.306691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:44123 (local address 127.2.243.129:44649)
0620 11:37:12.306894 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:12.306946 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:12.307416 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:12.307654 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:12.307658 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:12.308224 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:12.308412 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:12.308416 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.309092 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.309093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:12.309865 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:12.309867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.310211 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.310212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:12.310307 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:12.310815 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:12.310823 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:12.312256 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:12.313880 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:12.313885 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:12.313887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:12.314092 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:12.314267 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:12.314267 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:12.314267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:12.314359 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:12.314568 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:12.314572 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:12.314680 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 11:37:12.314759 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 11:37:12.314948 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":114,"threads_started":1}
I20260620 11:37:12.315452 3022 external_mini_cluster.cc:1428] Setting key c3da1ed6d970a158c74bd73643e7a137
I20260620 11:37:12.315711 22382 heartbeater.cc:347] Connected to a master server at 127.2.243.190:44123
I20260620 11:37:12.315835 22382 heartbeater.cc:464] Registering TS with master...
I20260620 11:37:12.316011 22382 heartbeater.cc:511] Master 127.2.243.190:44123 requested a full tablet report, sending...
I20260620 11:37:12.316572 22191 ts_manager.cc:194] Registered new tserver with Master: e6cc25cb87d946c0a1c91fa43e7dad59 (127.2.243.129:35931)
I20260620 11:37:12.317440 22191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.129@KRBTEST.COM'} at 127.2.243.129:44649
WARNING: no policy specified for kudu/127.2.243.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
Entry for principal kudu/127.2.243.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab.
I20260620 11:37:12.341454 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:44123
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:43285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:12.422489 22387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:12.422751 22387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:12.422792 22387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:12.424036 22387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:12.424116 22387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:12.424162 22387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:37:12.425635 22387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:43285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.130.keytab
--principal=kudu/127.2.243.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:12.425912 22387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:12.426153 22387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:12.431135 22395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:12.431135 22393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:12.431334 22387 server_base.cc:1034] running on GCE node
W20260620 11:37:12.431375 22392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:12.431644 22387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:12.431933 22387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:12.433162 22387 hybrid_clock.cc:648] HybridClock initialized: now 1781955432433158 us; error 50 us; skew 500 ppm
Jun 20 11:37:12 dist-test-slave-ctbw krb5kdc[22146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955432, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:12.435613 22387 init.cc:377] Logged in from keytab as kudu/127.2.243.130@KRBTEST.COM (short username kudu)
I20260620 11:37:12.436069 22387 webserver.cc:460] Webserver started at http://127.2.243.130:33247/ using document root <none> and password file <none>
I20260620 11:37:12.436298 22387 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:12.436347 22387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:12.436523 22387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:12.437559 22387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "3a10afd047fc4b0f82a940a1481e4411"
format_stamp: "Formatted at 2026-06-20 11:37:12 on dist-test-slave-ctbw"
server_key: "c01bc7fd9f6b4f8d5971c5e7165eeb99"
server_key_iv: "8e229bd11f7193a0b639a702b191fef7"
server_key_version: "encryptionkey@0"
I20260620 11:37:12.437923 22387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "3a10afd047fc4b0f82a940a1481e4411"
format_stamp: "Formatted at 2026-06-20 11:37:12 on dist-test-slave-ctbw"
server_key: "c01bc7fd9f6b4f8d5971c5e7165eeb99"
server_key_iv: "8e229bd11f7193a0b639a702b191fef7"
server_key_version: "encryptionkey@0"
I20260620 11:37:12.439298 22387 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 11:37:12.440056 22402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:12.440232 22387 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:37:12.440330 22387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "3a10afd047fc4b0f82a940a1481e4411"
format_stamp: "Formatted at 2026-06-20 11:37:12 on dist-test-slave-ctbw"
server_key: "c01bc7fd9f6b4f8d5971c5e7165eeb99"
server_key_iv: "8e229bd11f7193a0b639a702b191fef7"
server_key_version: "encryptionkey@0"
I20260620 11:37:12.440410 22387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:12.454106 22387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:12.456198 22387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:12.456312 22387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:12.456540 22387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:37:12.456831 22387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:37:12.456863 22387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:12.456887 22387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:37:12.456898 22387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:12.462852 22387 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:42637
I20260620 11:37:12.462996 22515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:42637 every 8 connection(s)
I20260620 11:37:12.463342 22387 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:37:12.466567 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 22387
I20260620 11:37:12.466703 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:37:12.466970 3022 external_mini_cluster.cc:1428] Setting key ea31edd7b54165a7735befcd3c74c1b3
Jun 20 11:37:12 dist-test-slave-ctbw krb5kdc[22146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955432, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.130@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:37:12.473767 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:12.464117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:37469 (local address 127.2.243.190:44123)
0620 11:37:12.464190 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:12.464192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:12.464370 (+ 178us) server_negotiation.cc:407] Connection header received
0620 11:37:12.464989 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:12.464990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:12.465000 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:12.465044 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:12.465956 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.466486 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.468028 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.468218 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.468275 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:12.470732 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:12.470734 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:12.470734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:12.470740 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:12.472260 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:12.472782 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:12.472783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:12.472783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:12.472843 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:12.473142 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:12.473143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:12.473143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:12.473241 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:12.473289 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:12.473638 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 11:37:12.473703 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 11:37:12.474048 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:12.464079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:44123 (local address 127.2.243.130:37469)
0620 11:37:12.464239 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:12.464280 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:12.464891 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:12.465099 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:12.465103 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:12.465652 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:12.465850 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:12.465856 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.466607 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.466608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:12.467944 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:12.467947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.468331 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.468332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:12.468488 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:12.469030 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:12.469040 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:12.470637 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:12.472352 (+ 1715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:12.472360 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:12.472363 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:12.472698 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:12.472916 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:12.472917 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:12.472917 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:12.473070 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:12.473310 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:12.473317 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:12.473494 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 11:37:12.473620 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 11:37:12.473804 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
I20260620 11:37:12.474629 22516 heartbeater.cc:347] Connected to a master server at 127.2.243.190:44123
I20260620 11:37:12.474737 22516 heartbeater.cc:464] Registering TS with master...
I20260620 11:37:12.474961 22516 heartbeater.cc:511] Master 127.2.243.190:44123 requested a full tablet report, sending...
I20260620 11:37:12.475370 22191 ts_manager.cc:194] Registered new tserver with Master: 3a10afd047fc4b0f82a940a1481e4411 (127.2.243.130:42637)
I20260620 11:37:12.475785 22191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.130@KRBTEST.COM'} at 127.2.243.130:37469
WARNING: no policy specified for kudu/127.2.243.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
Entry for principal kudu/127.2.243.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab.
I20260620 11:37:12.494436 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:44123
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:43285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:12.579813 22521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:12.580008 22521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:12.580031 22521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:12.581198 22521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:12.581240 22521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:12.581303 22521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:37:12.582732 22521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:43285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.131.keytab
--principal=kudu/127.2.243.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:44123
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:12.583012 22521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:12.583276 22521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:12.588212 22527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:12.588219 22526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:12.588217 22529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:12.588534 22521 server_base.cc:1034] running on GCE node
I20260620 11:37:12.588750 22521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:12.589015 22521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:12.590158 22521 hybrid_clock.cc:648] HybridClock initialized: now 1781955432590124 us; error 51 us; skew 500 ppm
Jun 20 11:37:12 dist-test-slave-ctbw krb5kdc[22146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955432, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:12.592669 22521 init.cc:377] Logged in from keytab as kudu/127.2.243.131@KRBTEST.COM (short username kudu)
I20260620 11:37:12.593096 22521 webserver.cc:460] Webserver started at http://127.2.243.131:39625/ using document root <none> and password file <none>
I20260620 11:37:12.593340 22521 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:12.593413 22521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:12.593559 22521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:12.594626 22521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "6d345a6c43d1459286a85e1b6bffe1ec"
format_stamp: "Formatted at 2026-06-20 11:37:12 on dist-test-slave-ctbw"
server_key: "88e9f416f2026a04ecb8bedc43abbd65"
server_key_iv: "1f491b4b14f38d1aa0077a785657079a"
server_key_version: "encryptionkey@0"
I20260620 11:37:12.595021 22521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "6d345a6c43d1459286a85e1b6bffe1ec"
format_stamp: "Formatted at 2026-06-20 11:37:12 on dist-test-slave-ctbw"
server_key: "88e9f416f2026a04ecb8bedc43abbd65"
server_key_iv: "1f491b4b14f38d1aa0077a785657079a"
server_key_version: "encryptionkey@0"
I20260620 11:37:12.596386 22521 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:37:12.597134 22536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:12.597339 22521 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:37:12.597424 22521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "6d345a6c43d1459286a85e1b6bffe1ec"
format_stamp: "Formatted at 2026-06-20 11:37:12 on dist-test-slave-ctbw"
server_key: "88e9f416f2026a04ecb8bedc43abbd65"
server_key_iv: "1f491b4b14f38d1aa0077a785657079a"
server_key_version: "encryptionkey@0"
I20260620 11:37:12.597509 22521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:12.605537 22521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:12.607364 22521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:12.607517 22521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:12.607721 22521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:37:12.608018 22521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:37:12.608075 22521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:12.608114 22521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:37:12.608140 22521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:12.613999 22521 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:45555
I20260620 11:37:12.614037 22649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:45555 every 8 connection(s)
I20260620 11:37:12.614363 22521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:37:12.619300 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 22521
I20260620 11:37:12.619445 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:37:12.619630 3022 external_mini_cluster.cc:1428] Setting key a2c3de3cd828402ec69294f66981974f
Jun 20 11:37:12 dist-test-slave-ctbw krb5kdc[22146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955432, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.131@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:37:12.623719 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:12.615068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:52365 (local address 127.2.243.190:44123)
0620 11:37:12.615121 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:12.615123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:12.615350 (+ 227us) server_negotiation.cc:407] Connection header received
0620 11:37:12.615884 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:12.615886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:12.615899 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:12.615955 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:12.616909 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.617413 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.618447 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.618659 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.618739 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:12.621279 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:12.621281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:12.621281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:12.621288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:12.622493 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:12.622938 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:12.622938 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:12.622939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:12.622998 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:12.623214 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:12.623215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:12.623215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:12.623316 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:12.623367 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:12.623595 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 11:37:12.623657 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 11:37:12.623991 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:12.615043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:44123 (local address 127.2.243.131:52365)
0620 11:37:12.615189 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:12.615233 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:12.615773 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:12.616022 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:12.616026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:12.616616 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:12.616797 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:12.616802 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.617515 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.617516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:12.618334 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:12.618336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.618746 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.618747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:12.618948 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:12.619686 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:12.619694 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:12.621195 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:12.622569 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:12.622575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:12.622576 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:12.622822 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:12.623058 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:12.623059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:12.623059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:12.623155 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:12.623372 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:12.623376 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:12.623485 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 11:37:12.623564 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 11:37:12.623678 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":77,"threads_started":1}
I20260620 11:37:12.624459 22650 heartbeater.cc:347] Connected to a master server at 127.2.243.190:44123
I20260620 11:37:12.624578 22650 heartbeater.cc:464] Registering TS with master...
I20260620 11:37:12.624794 22650 heartbeater.cc:511] Master 127.2.243.190:44123 requested a full tablet report, sending...
I20260620 11:37:12.625174 22191 ts_manager.cc:194] Registered new tserver with Master: 6d345a6c43d1459286a85e1b6bffe1ec (127.2.243.131:45555)
I20260620 11:37:12.625559 22191 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.2.243.131@KRBTEST.COM'} at 127.2.243.131:52365
I20260620 11:37:12.631261 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:37:12.638410 22231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:12.632742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.2.243.190:44123)
0620 11:37:12.632819 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:12.632821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:12.632952 (+ 131us) server_negotiation.cc:407] Connection header received
0620 11:37:12.632994 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:12.632995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:12.633003 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:12.633056 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:12.633669 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.634144 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.634853 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.635101 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.635155 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:12.636240 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:12.636241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:12.636242 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:12.636247 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:12.637465 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:12.637781 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:12.637782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:12.637782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:12.637830 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:12.637996 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:12.637997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:12.637997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:12.638123 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:12.638195 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:12.638298 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 11:37:12.638352 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 11:37:12.639940 22191 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:37:12.641453 22191 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:37:12.648136 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:12.643690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:35931 (local address 127.0.0.1:40314)
0620 11:37:12.644048 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:12.644059 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:12.644114 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:12.644469 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:12.644470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:12.644473 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:12.644518 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:12.644520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.645816 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.645817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:12.647064 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:12.647066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.647926 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.647927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:12.647978 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:12.647980 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:37:12.648038 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 11:37:12.648068 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":34,"thread_start_us":51,"threads_started":1}
I20260620 11:37:12.648157 22660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:12.643734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40222 (local address 127.2.243.130:42637)
0620 11:37:12.643920 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:12.643926 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:12.644036 (+ 110us) server_negotiation.cc:407] Connection header received
0620 11:37:12.644185 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:12.644188 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:12.644203 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:12.644271 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:12.644595 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.645697 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.646978 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.647695 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.647768 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:12.647833 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:12.648036 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 11:37:12.648076 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260620 11:37:12.648165 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:12.643879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40314 (local address 127.2.243.129:35931)
0620 11:37:12.644203 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:12.644207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:12.644216 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:37:12.644262 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:12.644264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:12.644281 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:12.644371 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:12.644585 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.645726 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.647151 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.647861 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.647928 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:12.647979 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:12.648051 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 11:37:12.648089 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260620 11:37:12.648247 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:12.644048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.2.243.131:45555)
0620 11:37:12.644649 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:12.644655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:12.644668 (+ 13us) server_negotiation.cc:407] Connection header received
0620 11:37:12.644699 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:12.644700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:12.644711 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:12.644809 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:12.644908 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.645576 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.647161 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.647655 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.647768 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:12.647829 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:12.648111 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 11:37:12.648156 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":76,"threads_started":1}
I20260620 11:37:12.648357 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:12.643644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:42637 (local address 127.0.0.1:40222)
0620 11:37:12.643930 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:12.643973 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:12.644080 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:12.644418 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:12.644421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:12.644443 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:12.644507 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:12.644509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.645819 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.645820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:12.646912 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:12.646915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.647872 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.647873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:12.647937 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:12.647941 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:37:12.648235 (+ 294us) client_negotiation.cc:240] Negotiation successful
0620 11:37:12.648288 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":22,"thread_start_us":66,"threads_started":1}
I20260620 11:37:12.648072 22661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:12.643881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:45555 (local address 127.0.0.1:39152)
0620 11:37:12.644049 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:12.644058 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:12.644099 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:12.644807 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:12.644808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:12.644810 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:12.644851 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:12.644853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.645683 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.645684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:12.647062 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:12.647066 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.647768 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.647769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:12.647843 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:12.647856 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 11:37:12.647921 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 11:37:12.647973 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":32,"thread_start_us":43,"threads_started":1}
I20260620 11:37:12.649448 22316 tablet_service.cc:1467] Processing CreateTablet for tablet c8a63dbe6add43d5b774be2219fe65e7 (DEFAULT_TABLE table=test-table [id=913cc6b0c4db4180a1505a4afb96eec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:37:12.649808 22316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8a63dbe6add43d5b774be2219fe65e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:12.649744 22584 tablet_service.cc:1467] Processing CreateTablet for tablet c8a63dbe6add43d5b774be2219fe65e7 (DEFAULT_TABLE table=test-table [id=913cc6b0c4db4180a1505a4afb96eec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:37:12.650040 22584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8a63dbe6add43d5b774be2219fe65e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:12.653231 22664 tablet_bootstrap.cc:492] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59: Bootstrap starting.
I20260620 11:37:12.653591 22450 tablet_service.cc:1467] Processing CreateTablet for tablet c8a63dbe6add43d5b774be2219fe65e7 (DEFAULT_TABLE table=test-table [id=913cc6b0c4db4180a1505a4afb96eec5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:37:12.653878 22664 tablet_bootstrap.cc:654] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:12.653918 22450 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8a63dbe6add43d5b774be2219fe65e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:12.654367 22664 log.cc:824] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:12.654726 22665 tablet_bootstrap.cc:492] T c8a63dbe6add43d5b774be2219fe65e7 P 6d345a6c43d1459286a85e1b6bffe1ec: Bootstrap starting.
I20260620 11:37:12.655423 22665 tablet_bootstrap.cc:654] T c8a63dbe6add43d5b774be2219fe65e7 P 6d345a6c43d1459286a85e1b6bffe1ec: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:12.655870 22665 log.cc:824] T c8a63dbe6add43d5b774be2219fe65e7 P 6d345a6c43d1459286a85e1b6bffe1ec: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:12.656356 22664 tablet_bootstrap.cc:492] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59: No bootstrap required, opened a new log
I20260620 11:37:12.656424 22664 ts_tablet_manager.cc:1397] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 11:37:12.656590 22665 tablet_bootstrap.cc:492] T c8a63dbe6add43d5b774be2219fe65e7 P 6d345a6c43d1459286a85e1b6bffe1ec: No bootstrap required, opened a new log
I20260620 11:37:12.656648 22665 ts_tablet_manager.cc:1397] T c8a63dbe6add43d5b774be2219fe65e7 P 6d345a6c43d1459286a85e1b6bffe1ec: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:37:12.656982 22668 tablet_bootstrap.cc:492] T c8a63dbe6add43d5b774be2219fe65e7 P 3a10afd047fc4b0f82a940a1481e4411: Bootstrap starting.
I20260620 11:37:12.657559 22668 tablet_bootstrap.cc:654] T c8a63dbe6add43d5b774be2219fe65e7 P 3a10afd047fc4b0f82a940a1481e4411: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:12.658046 22668 log.cc:824] T c8a63dbe6add43d5b774be2219fe65e7 P 3a10afd047fc4b0f82a940a1481e4411: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:12.658427 22664 raft_consensus.cc:348] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a10afd047fc4b0f82a940a1481e4411" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42637 } } peers { permanent_uuid: "6d345a6c43d1459286a85e1b6bffe1ec" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45555 } } peers { permanent_uuid: "e6cc25cb87d946c0a1c91fa43e7dad59" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 35931 } }
I20260620 11:37:12.658524 22665 raft_consensus.cc:348] T c8a63dbe6add43d5b774be2219fe65e7 P 6d345a6c43d1459286a85e1b6bffe1ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a10afd047fc4b0f82a940a1481e4411" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42637 } } peers { permanent_uuid: "6d345a6c43d1459286a85e1b6bffe1ec" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45555 } } peers { permanent_uuid: "e6cc25cb87d946c0a1c91fa43e7dad59" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 35931 } }
I20260620 11:37:12.658614 22664 raft_consensus.cc:374] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:12.658672 22664 raft_consensus.cc:729] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6cc25cb87d946c0a1c91fa43e7dad59, State: Initialized, Role: FOLLOWER
I20260620 11:37:12.658674 22665 raft_consensus.cc:374] T c8a63dbe6add43d5b774be2219fe65e7 P 6d345a6c43d1459286a85e1b6bffe1ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:12.658718 22665 raft_consensus.cc:729] T c8a63dbe6add43d5b774be2219fe65e7 P 6d345a6c43d1459286a85e1b6bffe1ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d345a6c43d1459286a85e1b6bffe1ec, State: Initialized, Role: FOLLOWER
I20260620 11:37:12.658763 22668 tablet_bootstrap.cc:492] T c8a63dbe6add43d5b774be2219fe65e7 P 3a10afd047fc4b0f82a940a1481e4411: No bootstrap required, opened a new log
I20260620 11:37:12.658833 22668 ts_tablet_manager.cc:1397] T c8a63dbe6add43d5b774be2219fe65e7 P 3a10afd047fc4b0f82a940a1481e4411: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:37:12.658813 22664 consensus_queue.cc:260] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a10afd047fc4b0f82a940a1481e4411" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42637 } } peers { permanent_uuid: "6d345a6c43d1459286a85e1b6bffe1ec" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45555 } } peers { permanent_uuid: "e6cc25cb87d946c0a1c91fa43e7dad59" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 35931 } }
I20260620 11:37:12.658843 22665 consensus_queue.cc:260] T c8a63dbe6add43d5b774be2219fe65e7 P 6d345a6c43d1459286a85e1b6bffe1ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a10afd047fc4b0f82a940a1481e4411" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42637 } } peers { permanent_uuid: "6d345a6c43d1459286a85e1b6bffe1ec" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45555 } } peers { permanent_uuid: "e6cc25cb87d946c0a1c91fa43e7dad59" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 35931 } }
I20260620 11:37:12.659128 22664 ts_tablet_manager.cc:1428] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:37:12.659267 22382 heartbeater.cc:503] Master 127.2.243.190:44123 was elected leader, sending a full tablet report...
I20260620 11:37:12.659127 22665 ts_tablet_manager.cc:1428] T c8a63dbe6add43d5b774be2219fe65e7 P 6d345a6c43d1459286a85e1b6bffe1ec: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260620 11:37:12.659148 22650 heartbeater.cc:503] Master 127.2.243.190:44123 was elected leader, sending a full tablet report...
I20260620 11:37:12.660979 22668 raft_consensus.cc:348] T c8a63dbe6add43d5b774be2219fe65e7 P 3a10afd047fc4b0f82a940a1481e4411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a10afd047fc4b0f82a940a1481e4411" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42637 } } peers { permanent_uuid: "6d345a6c43d1459286a85e1b6bffe1ec" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45555 } } peers { permanent_uuid: "e6cc25cb87d946c0a1c91fa43e7dad59" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 35931 } }
I20260620 11:37:12.661135 22668 raft_consensus.cc:374] T c8a63dbe6add43d5b774be2219fe65e7 P 3a10afd047fc4b0f82a940a1481e4411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:12.661181 22668 raft_consensus.cc:729] T c8a63dbe6add43d5b774be2219fe65e7 P 3a10afd047fc4b0f82a940a1481e4411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a10afd047fc4b0f82a940a1481e4411, State: Initialized, Role: FOLLOWER
I20260620 11:37:12.661288 22668 consensus_queue.cc:260] T c8a63dbe6add43d5b774be2219fe65e7 P 3a10afd047fc4b0f82a940a1481e4411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a10afd047fc4b0f82a940a1481e4411" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42637 } } peers { permanent_uuid: "6d345a6c43d1459286a85e1b6bffe1ec" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45555 } } peers { permanent_uuid: "e6cc25cb87d946c0a1c91fa43e7dad59" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 35931 } }
I20260620 11:37:12.661540 22668 ts_tablet_manager.cc:1428] T c8a63dbe6add43d5b774be2219fe65e7 P 3a10afd047fc4b0f82a940a1481e4411: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260620 11:37:12.661569 22516 heartbeater.cc:503] Master 127.2.243.190:44123 was elected leader, sending a full tablet report...
W20260620 11:37:12.713968 22517 tablet.cc:2369] T c8a63dbe6add43d5b774be2219fe65e7 P 3a10afd047fc4b0f82a940a1481e4411: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 11:37:12.806757 22383 tablet.cc:2369] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:37:12.854534 22670 raft_consensus.cc:482] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:37:12.854701 22670 raft_consensus.cc:504] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a10afd047fc4b0f82a940a1481e4411" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42637 } } peers { permanent_uuid: "6d345a6c43d1459286a85e1b6bffe1ec" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45555 } } peers { permanent_uuid: "e6cc25cb87d946c0a1c91fa43e7dad59" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 35931 } }
I20260620 11:37:12.855127 22670 leader_election.cc:288] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a10afd047fc4b0f82a940a1481e4411 (127.2.243.130:42637), 6d345a6c43d1459286a85e1b6bffe1ec (127.2.243.131:45555)
I20260620 11:37:12.858552 22660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:12.855515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:48755 (local address 127.2.243.130:42637)
0620 11:37:12.855593 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:12.855595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:12.855695 (+ 100us) server_negotiation.cc:407] Connection header received
0620 11:37:12.855835 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:12.855837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:12.855847 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:12.855943 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:12.856156 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.856680 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.857810 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.858165 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.858222 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:12.858229 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:12.858436 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 11:37:12.858480 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 11:37:12.858577 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:12.855729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:45555 (local address 127.2.243.129:42711)
0620 11:37:12.855840 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:12.855851 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:12.855915 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:12.856097 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:12.856098 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:12.856099 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:12.856192 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:12.856195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.856849 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.856849 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:12.857663 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:12.857665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.858308 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.858309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:12.858361 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:12.858365 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:37:12.858441 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 11:37:12.858480 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":2,"thread_start_us":56,"threads_started":1}
I20260620 11:37:12.858573 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:12.855782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:42711 (local address 127.2.243.131:45555)
0620 11:37:12.855880 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:12.855882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:12.855892 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:37:12.856002 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:12.856003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:12.856013 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:12.856086 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:12.856272 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.856757 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.857764 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.858154 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.858280 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:12.858289 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:12.858467 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 11:37:12.858503 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 11:37:12.858588 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:12.855430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:42637 (local address 127.2.243.129:48755)
0620 11:37:12.855592 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:12.855605 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:12.855679 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:12.855978 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:12.855979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:12.855982 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:12.856069 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:12.856071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.856777 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.856778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:12.857673 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:12.857675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.858264 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.858265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:12.858339 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:12.858343 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:37:12.858404 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 11:37:12.858485 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":48,"thread_start_us":106,"threads_started":1}
I20260620 11:37:12.859006 22470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a63dbe6add43d5b774be2219fe65e7" candidate_uuid: "e6cc25cb87d946c0a1c91fa43e7dad59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a10afd047fc4b0f82a940a1481e4411" is_pre_election: true
I20260620 11:37:12.859018 22604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a63dbe6add43d5b774be2219fe65e7" candidate_uuid: "e6cc25cb87d946c0a1c91fa43e7dad59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d345a6c43d1459286a85e1b6bffe1ec" is_pre_election: true
I20260620 11:37:12.859144 22470 raft_consensus.cc:2454] T c8a63dbe6add43d5b774be2219fe65e7 P 3a10afd047fc4b0f82a940a1481e4411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6cc25cb87d946c0a1c91fa43e7dad59 in term 0.
I20260620 11:37:12.859177 22604 raft_consensus.cc:2454] T c8a63dbe6add43d5b774be2219fe65e7 P 6d345a6c43d1459286a85e1b6bffe1ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6cc25cb87d946c0a1c91fa43e7dad59 in term 0.
I20260620 11:37:12.859416 22270 leader_election.cc:302] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a10afd047fc4b0f82a940a1481e4411, e6cc25cb87d946c0a1c91fa43e7dad59; no voters:
I20260620 11:37:12.859599 22670 raft_consensus.cc:2781] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:37:12.859715 22670 raft_consensus.cc:482] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:37:12.859758 22670 raft_consensus.cc:3037] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:12.860502 22670 raft_consensus.cc:504] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a10afd047fc4b0f82a940a1481e4411" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42637 } } peers { permanent_uuid: "6d345a6c43d1459286a85e1b6bffe1ec" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45555 } } peers { permanent_uuid: "e6cc25cb87d946c0a1c91fa43e7dad59" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 35931 } }
I20260620 11:37:12.860647 22670 leader_election.cc:288] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59 [CANDIDATE]: Term 1 election: Requested vote from peers 3a10afd047fc4b0f82a940a1481e4411 (127.2.243.130:42637), 6d345a6c43d1459286a85e1b6bffe1ec (127.2.243.131:45555)
I20260620 11:37:12.860848 22604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a63dbe6add43d5b774be2219fe65e7" candidate_uuid: "e6cc25cb87d946c0a1c91fa43e7dad59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d345a6c43d1459286a85e1b6bffe1ec"
I20260620 11:37:12.860850 22470 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a63dbe6add43d5b774be2219fe65e7" candidate_uuid: "e6cc25cb87d946c0a1c91fa43e7dad59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a10afd047fc4b0f82a940a1481e4411"
I20260620 11:37:12.860919 22604 raft_consensus.cc:3037] T c8a63dbe6add43d5b774be2219fe65e7 P 6d345a6c43d1459286a85e1b6bffe1ec [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:12.860960 22470 raft_consensus.cc:3037] T c8a63dbe6add43d5b774be2219fe65e7 P 3a10afd047fc4b0f82a940a1481e4411 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:12.861624 22604 raft_consensus.cc:2454] T c8a63dbe6add43d5b774be2219fe65e7 P 6d345a6c43d1459286a85e1b6bffe1ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6cc25cb87d946c0a1c91fa43e7dad59 in term 1.
I20260620 11:37:12.861752 22470 raft_consensus.cc:2454] T c8a63dbe6add43d5b774be2219fe65e7 P 3a10afd047fc4b0f82a940a1481e4411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6cc25cb87d946c0a1c91fa43e7dad59 in term 1.
I20260620 11:37:12.861790 22270 leader_election.cc:302] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d345a6c43d1459286a85e1b6bffe1ec, e6cc25cb87d946c0a1c91fa43e7dad59; no voters:
I20260620 11:37:12.861876 22670 raft_consensus.cc:2781] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:12.862030 22670 raft_consensus.cc:686] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59 [term 1 LEADER]: Becoming Leader. State: Replica: e6cc25cb87d946c0a1c91fa43e7dad59, State: Running, Role: LEADER
I20260620 11:37:12.862135 22670 consensus_queue.cc:237] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a10afd047fc4b0f82a940a1481e4411" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42637 } } peers { permanent_uuid: "6d345a6c43d1459286a85e1b6bffe1ec" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45555 } } peers { permanent_uuid: "e6cc25cb87d946c0a1c91fa43e7dad59" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 35931 } }
I20260620 11:37:12.862985 22190 catalog_manager.cc:5495] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6cc25cb87d946c0a1c91fa43e7dad59 (127.2.243.129). New cstate: current_term: 1 leader_uuid: "e6cc25cb87d946c0a1c91fa43e7dad59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a10afd047fc4b0f82a940a1481e4411" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d345a6c43d1459286a85e1b6bffe1ec" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6cc25cb87d946c0a1c91fa43e7dad59" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 35931 } health_report { overall_health: HEALTHY } } }
W20260620 11:37:12.864940 22651 tablet.cc:2369] T c8a63dbe6add43d5b774be2219fe65e7 P 6d345a6c43d1459286a85e1b6bffe1ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:37:12.873951 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:12.871864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40320 (local address 127.2.243.129:35931)
0620 11:37:12.871921 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:12.871923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:12.871930 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:37:12.871957 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:12.871958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:12.871973 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:12.872030 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:12.872176 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.872621 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.873305 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:12.873475 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:12.873523 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:12.873646 (+ 123us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:12.873735 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:12.873791 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:12.873831 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 11:37:12.873847 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 11:37:12.875845 22470 raft_consensus.cc:1261] T c8a63dbe6add43d5b774be2219fe65e7 P 3a10afd047fc4b0f82a940a1481e4411 [term 1 FOLLOWER]: Refusing update from remote peer e6cc25cb87d946c0a1c91fa43e7dad59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:37:12.875865 22604 raft_consensus.cc:1261] T c8a63dbe6add43d5b774be2219fe65e7 P 6d345a6c43d1459286a85e1b6bffe1ec [term 1 FOLLOWER]: Refusing update from remote peer e6cc25cb87d946c0a1c91fa43e7dad59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:37:12.876181 22670 consensus_queue.cc:1035] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a10afd047fc4b0f82a940a1481e4411" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:37:12.876369 22675 consensus_queue.cc:1035] T c8a63dbe6add43d5b774be2219fe65e7 P e6cc25cb87d946c0a1c91fa43e7dad59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d345a6c43d1459286a85e1b6bffe1ec" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 45555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:37:12.878647 22679 mvcc.cc:205] Tried to move back new op lower bound from 7298889453056413696 to 7298889453003968512. Current Snapshot: MvccSnapshot[applied={T|T < 7298889453056413696}]
I20260620 11:37:12.879021 22678 mvcc.cc:205] Tried to move back new op lower bound from 7298889453056413696 to 7298889453003968512. Current Snapshot: MvccSnapshot[applied={T|T < 7298889453056413696 or (T in {7298889453056413696})}]
I20260620 11:37:12.880669 22190 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:37:12.880767 22190 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:37:12.881506 22190 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f104c2b8bb764e7d9f787ecff741fc6d: Sending DeleteTablet for 3 replicas of tablet c8a63dbe6add43d5b774be2219fe65e7
I20260620 11:37:12.881844 22450 tablet_service.cc:1514] Processing DeleteTablet for tablet c8a63dbe6add43d5b774be2219fe65e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:37:12 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:40222
I20260620 11:37:12.881857 22316 tablet_service.cc:1514] Processing DeleteTablet for tablet c8a63dbe6add43d5b774be2219fe65e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:37:12 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:40314
I20260620 11:37:12.882051 22687 tablet_replica.cc:330] T c8a63dbe6add43d5b774be2219fe65e7 P 3a10afd047fc4b0f82a940a1481e4411: stopping tablet replica
I20260620 11:37:12.882076 22584 tablet_service.cc:1514] Processing DeleteTablet for tablet c8a63dbe6add43d5b774be2219fe65e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:37:12 UTC) from {username='kudu', principal='kudu/127.2.243.190@KRBTEST.COM'} at 127.0.0.1:39152
I20260620 11:37:12.882146 22687 raft_consensus.cc:2229] T c8a63dbe6add43d5b774be2219fe65e7 P 3a10afd047fc4b0f82a940a1481e4411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:37:12.882241 22687 raft_consensus.cc:2258] T c8a63dbe6add43d5b774be2219fe65e7 P 3a10afd047fc4b0f82a940a1481e4411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:37:12.882576 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 22253
I20260620 11:37:12.883252 22687 ts_tablet_manager.cc:1905] T c8a63dbe6add43d5b774be2219fe65e7 P 3a10afd047fc4b0f82a940a1481e4411: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:37:12.883291 22689 tablet_replica.cc:330] T c8a63dbe6add43d5b774be2219fe65e7 P 6d345a6c43d1459286a85e1b6bffe1ec: stopping tablet replica
I20260620 11:37:12.883375 22689 raft_consensus.cc:2229] T c8a63dbe6add43d5b774be2219fe65e7 P 6d345a6c43d1459286a85e1b6bffe1ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:37:12.883468 22689 raft_consensus.cc:2258] T c8a63dbe6add43d5b774be2219fe65e7 P 6d345a6c43d1459286a85e1b6bffe1ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:37:12.884325 22689 ts_tablet_manager.cc:1905] T c8a63dbe6add43d5b774be2219fe65e7 P 6d345a6c43d1459286a85e1b6bffe1ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:37:12.885232 22687 ts_tablet_manager.cc:1918] T c8a63dbe6add43d5b774be2219fe65e7 P 3a10afd047fc4b0f82a940a1481e4411: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:37:12.885324 22687 log.cc:1196] T c8a63dbe6add43d5b774be2219fe65e7 P 3a10afd047fc4b0f82a940a1481e4411: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-1/wal/wals/c8a63dbe6add43d5b774be2219fe65e7
I20260620 11:37:12.885589 22687 ts_tablet_manager.cc:1939] T c8a63dbe6add43d5b774be2219fe65e7 P 3a10afd047fc4b0f82a940a1481e4411: Deleting consensus metadata
I20260620 11:37:12.885895 22689 ts_tablet_manager.cc:1918] T c8a63dbe6add43d5b774be2219fe65e7 P 6d345a6c43d1459286a85e1b6bffe1ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:37:12.885983 22689 log.cc:1196] T c8a63dbe6add43d5b774be2219fe65e7 P 6d345a6c43d1459286a85e1b6bffe1ec: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/c8a63dbe6add43d5b774be2219fe65e7
I20260620 11:37:12.886173 22176 catalog_manager.cc:4841] TS 3a10afd047fc4b0f82a940a1481e4411 (127.2.243.130:42637): tablet c8a63dbe6add43d5b774be2219fe65e7 (table test-table [id=913cc6b0c4db4180a1505a4afb96eec5]) successfully deleted
I20260620 11:37:12.886298 22689 ts_tablet_manager.cc:1939] T c8a63dbe6add43d5b774be2219fe65e7 P 6d345a6c43d1459286a85e1b6bffe1ec: Deleting consensus metadata
I20260620 11:37:12.886675 22176 catalog_manager.cc:4841] TS 6d345a6c43d1459286a85e1b6bffe1ec (127.2.243.131:45555): tablet c8a63dbe6add43d5b774be2219fe65e7 (table test-table [id=913cc6b0c4db4180a1505a4afb96eec5]) successfully deleted
W20260620 11:37:12.888518 22177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.243.129:35931 (error 108)
I20260620 11:37:12.888754 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 22387
I20260620 11:37:12.889344 22661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:12.888883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:35931 (local address 127.0.0.1:40336)
0620 11:37:12.889268 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:12.889305 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.129:35931: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363}
W20260620 11:37:12.889461 22177 catalog_manager.cc:4584] TS e6cc25cb87d946c0a1c91fa43e7dad59 (127.2.243.129:35931): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c8a63dbe6add43d5b774be2219fe65e7: Network error: Client connection negotiation failed: client connection to 127.2.243.129:35931: connect: Connection refused (error 111)
I20260620 11:37:12.893631 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 22521
I20260620 11:37:12.897981 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 22159
2026-06-20T11:37:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2071 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T11:37:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:12Z Disabled control of system clock
I20260620 11:37:12.925328 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:46251
--webserver_interface=127.2.243.190
--webserver_port=0
--builtin_ntp_servers=127.2.243.148:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:46251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 11:37:13.005029 22694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:13.005239 22694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:13.005259 22694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:13.006513 22694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:13.006557 22694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:13.006570 22694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:13.006583 22694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:13.006644 22694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:13.007985 22694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:46251
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:46251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:13.008248 22694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:13.008551 22694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:13.013779 22700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:13.013921 22694 server_base.cc:1034] running on GCE node
W20260620 11:37:13.013761 22699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:13.013761 22702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:13.014304 22694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:13.014734 22694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:13.015944 22694 hybrid_clock.cc:648] HybridClock initialized: now 1781955433015929 us; error 28 us; skew 500 ppm
I20260620 11:37:13.017234 22694 webserver.cc:460] Webserver started at http://127.2.243.190:32879/ using document root <none> and password file <none>
I20260620 11:37:13.017550 22694 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:13.017640 22694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:13.017884 22694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:13.019202 22694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "f36cf3a2e1d24b0a93fd2e0dde346cdd"
format_stamp: "Formatted at 2026-06-20 11:37:13 on dist-test-slave-ctbw"
server_key: "fcb5d91f0e5cd931f313d6867d1c807f"
server_key_iv: "000d89c734b0283ea708770c24ce500f"
server_key_version: "encryptionkey@0"
I20260620 11:37:13.019596 22694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "f36cf3a2e1d24b0a93fd2e0dde346cdd"
format_stamp: "Formatted at 2026-06-20 11:37:13 on dist-test-slave-ctbw"
server_key: "fcb5d91f0e5cd931f313d6867d1c807f"
server_key_iv: "000d89c734b0283ea708770c24ce500f"
server_key_version: "encryptionkey@0"
I20260620 11:37:13.021049 22694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:37:13.022009 22708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:13.022342 22694 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:37:13.022446 22694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "f36cf3a2e1d24b0a93fd2e0dde346cdd"
format_stamp: "Formatted at 2026-06-20 11:37:13 on dist-test-slave-ctbw"
server_key: "fcb5d91f0e5cd931f313d6867d1c807f"
server_key_iv: "000d89c734b0283ea708770c24ce500f"
server_key_version: "encryptionkey@0"
I20260620 11:37:13.022535 22694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:13.066699 22694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:13.067116 22694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:13.067312 22694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:13.071595 22694 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:46251
I20260620 11:37:13.071651 22760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:46251 every 8 connection(s)
I20260620 11:37:13.072041 22694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:13.073073 22761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:13.075510 22761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd: Bootstrap starting.
I20260620 11:37:13.076175 22761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:13.076491 22761 log.cc:824] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:13.077333 22761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd: No bootstrap required, opened a new log
I20260620 11:37:13.079063 22761 raft_consensus.cc:348] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36cf3a2e1d24b0a93fd2e0dde346cdd" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46251 } }
I20260620 11:37:13.079210 22761 raft_consensus.cc:374] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:13.079236 22761 raft_consensus.cc:729] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f36cf3a2e1d24b0a93fd2e0dde346cdd, State: Initialized, Role: FOLLOWER
I20260620 11:37:13.079408 22761 consensus_queue.cc:260] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36cf3a2e1d24b0a93fd2e0dde346cdd" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46251 } }
I20260620 11:37:13.079473 22761 raft_consensus.cc:388] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:13.079515 22761 raft_consensus.cc:482] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:13.079564 22761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:13.080089 22761 raft_consensus.cc:504] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36cf3a2e1d24b0a93fd2e0dde346cdd" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46251 } }
I20260620 11:37:13.080178 22761 leader_election.cc:302] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f36cf3a2e1d24b0a93fd2e0dde346cdd; no voters:
I20260620 11:37:13.080190 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 22694
I20260620 11:37:13.080332 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:13.080396 22761 leader_election.cc:288] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:13.080423 22764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:13.080552 22764 raft_consensus.cc:686] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd [term 1 LEADER]: Becoming Leader. State: Replica: f36cf3a2e1d24b0a93fd2e0dde346cdd, State: Running, Role: LEADER
I20260620 11:37:13.080569 3022 external_mini_cluster.cc:1428] Setting key d69ff3352476f31bd939fcac5736aa55
I20260620 11:37:13.080673 22761 sys_catalog.cc:564] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:13.080647 22764 consensus_queue.cc:237] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36cf3a2e1d24b0a93fd2e0dde346cdd" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46251 } }
I20260620 11:37:13.080994 22764 sys_catalog.cc:455] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f36cf3a2e1d24b0a93fd2e0dde346cdd. Latest consensus state: current_term: 1 leader_uuid: "f36cf3a2e1d24b0a93fd2e0dde346cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36cf3a2e1d24b0a93fd2e0dde346cdd" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46251 } } }
I20260620 11:37:13.081073 22764 sys_catalog.cc:458] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:13.081230 22765 sys_catalog.cc:455] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f36cf3a2e1d24b0a93fd2e0dde346cdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f36cf3a2e1d24b0a93fd2e0dde346cdd" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46251 } } }
I20260620 11:37:13.081377 22765 sys_catalog.cc:458] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:13.081633 22771 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:13.082156 22771 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:13.083823 22771 catalog_manager.cc:1347] Generated new cluster ID: 4114221d484f470882e8dea9499d9270
I20260620 11:37:13.083875 22771 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:37:13.087455 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:13.081176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41104 (local address 127.2.243.190:46251)
0620 11:37:13.081871 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:13.081883 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:13.081921 (+ 38us) server_negotiation.cc:407] Connection header received
0620 11:37:13.082051 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:13.082059 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:13.082147 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:13.082307 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:13.082627 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.084001 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.085125 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.085703 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.085817 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:13.086592 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:13.086596 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:13.086596 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:13.086813 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:13.086857 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:13.086899 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:13.087131 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:13.087167 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 11:37:13.087246 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":107,"threads_started":1}
I20260620 11:37:13.106215 22771 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:13.106968 22771 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:13.114332 22771 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd: Generated new TSK 0
I20260620 11:37:13.114529 22771 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:37:13.117929 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.129:0
--local_ip_for_outbound_sockets=127.2.243.129
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:46251
--builtin_ntp_servers=127.2.243.148:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 11:37:13.199318 22785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:13.199548 22785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:13.199584 22785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:13.200830 22785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:13.200901 22785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:13.200944 22785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.129
I20260620 11:37:13.202327 22785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.2.243.129
--webserver_port=0
--tserver_master_addrs=127.2.243.190:46251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.129
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:13.202600 22785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:13.202953 22785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:13.208796 22793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:13.208791 22791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:13.208796 22790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:13.209108 22785 server_base.cc:1034] running on GCE node
I20260620 11:37:13.209339 22785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:13.209601 22785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:13.210788 22785 hybrid_clock.cc:648] HybridClock initialized: now 1781955433210777 us; error 31 us; skew 500 ppm
I20260620 11:37:13.212051 22785 webserver.cc:460] Webserver started at http://127.2.243.129:44301/ using document root <none> and password file <none>
I20260620 11:37:13.212352 22785 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:13.212428 22785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:13.212566 22785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:13.213635 22785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-0/data/instance:
uuid: "dfcc4c45e2534128b5fce461b90740b0"
format_stamp: "Formatted at 2026-06-20 11:37:13 on dist-test-slave-ctbw"
server_key: "d903f5086e230e5929d77af48c9d5d8e"
server_key_iv: "12e83d4b9ff2ccd1e3ca97f9ea18968b"
server_key_version: "encryptionkey@0"
I20260620 11:37:13.214003 22785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance:
uuid: "dfcc4c45e2534128b5fce461b90740b0"
format_stamp: "Formatted at 2026-06-20 11:37:13 on dist-test-slave-ctbw"
server_key: "d903f5086e230e5929d77af48c9d5d8e"
server_key_iv: "12e83d4b9ff2ccd1e3ca97f9ea18968b"
server_key_version: "encryptionkey@0"
I20260620 11:37:13.215400 22785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 11:37:13.216140 22799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:13.216344 22785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:37:13.216447 22785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-0/wal
uuid: "dfcc4c45e2534128b5fce461b90740b0"
format_stamp: "Formatted at 2026-06-20 11:37:13 on dist-test-slave-ctbw"
server_key: "d903f5086e230e5929d77af48c9d5d8e"
server_key_iv: "12e83d4b9ff2ccd1e3ca97f9ea18968b"
server_key_version: "encryptionkey@0"
I20260620 11:37:13.216532 22785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:13.226398 22785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:13.226688 22785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:13.226840 22785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:13.227129 22785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:37:13.227451 22785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:37:13.227509 22785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:13.227543 22785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:37:13.227569 22785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:13.233530 22785 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.129:44101
I20260620 11:37:13.233579 22912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.129:44101 every 8 connection(s)
I20260620 11:37:13.233886 22785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-0/data/info.pb
I20260620 11:37:13.238796 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:13.234588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.129:35611 (local address 127.2.243.190:46251)
0620 11:37:13.234635 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:13.234637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:13.234906 (+ 269us) server_negotiation.cc:407] Connection header received
0620 11:37:13.235398 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:13.235399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:13.235412 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:13.235480 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:13.236138 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.236624 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.237573 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.237794 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.237846 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:13.238581 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:13.238582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:13.238582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:13.238586 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:13.238596 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:13.238600 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:13.238632 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:13.238699 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 11:37:13.238740 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:37:13.238991 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:13.234553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:46251 (local address 127.2.243.129:35611)
0620 11:37:13.234738 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:13.234779 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:13.235305 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:13.235533 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:13.235537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:13.235545 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:13.236025 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:13.236029 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.236710 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.236710 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:13.237497 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:13.237499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.237870 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.237871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:13.237975 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:13.238462 (+ 487us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:37:13.238463 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:13.238490 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:37:13.238510 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:13.238640 (+ 130us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:13.238643 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:13.238645 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:37:13.238691 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 11:37:13.238763 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
I20260620 11:37:13.239542 22913 heartbeater.cc:347] Connected to a master server at 127.2.243.190:46251
I20260620 11:37:13.239646 22913 heartbeater.cc:464] Registering TS with master...
I20260620 11:37:13.239832 22913 heartbeater.cc:511] Master 127.2.243.190:46251 requested a full tablet report, sending...
I20260620 11:37:13.240319 22725 ts_manager.cc:194] Registered new tserver with Master: dfcc4c45e2534128b5fce461b90740b0 (127.2.243.129:44101)
I20260620 11:37:13.240888 22725 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.243.129:35611
I20260620 11:37:13.243041 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 22785
I20260620 11:37:13.243119 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-0/wal/instance
I20260620 11:37:13.243268 3022 external_mini_cluster.cc:1428] Setting key f329df224409247303fd50dea6b777a4
I20260620 11:37:13.244134 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.130:0
--local_ip_for_outbound_sockets=127.2.243.130
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:46251
--builtin_ntp_servers=127.2.243.148:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 11:37:13.326460 22916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:13.326674 22916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:13.326697 22916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:13.327924 22916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:13.327965 22916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:13.327989 22916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.130
I20260620 11:37:13.329231 22916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.2.243.130
--webserver_port=0
--tserver_master_addrs=127.2.243.190:46251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.130
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:13.329407 22916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:13.329638 22916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:13.334440 22924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:13.334446 22922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:13.334525 22921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:13.334806 22916 server_base.cc:1034] running on GCE node
I20260620 11:37:13.335089 22916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:13.335307 22916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:13.336474 22916 hybrid_clock.cc:648] HybridClock initialized: now 1781955433336452 us; error 36 us; skew 500 ppm
I20260620 11:37:13.337796 22916 webserver.cc:460] Webserver started at http://127.2.243.130:46545/ using document root <none> and password file <none>
I20260620 11:37:13.338001 22916 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:13.338047 22916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:13.338165 22916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:13.339325 22916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-1/data/instance:
uuid: "4b3cea17d1d14678b3303b28ebe584e9"
format_stamp: "Formatted at 2026-06-20 11:37:13 on dist-test-slave-ctbw"
server_key: "58a018a326a0cf22beffd6aa26026292"
server_key_iv: "b7d68ccb36e26141d0597799821433d0"
server_key_version: "encryptionkey@0"
I20260620 11:37:13.339706 22916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance:
uuid: "4b3cea17d1d14678b3303b28ebe584e9"
format_stamp: "Formatted at 2026-06-20 11:37:13 on dist-test-slave-ctbw"
server_key: "58a018a326a0cf22beffd6aa26026292"
server_key_iv: "b7d68ccb36e26141d0597799821433d0"
server_key_version: "encryptionkey@0"
I20260620 11:37:13.341019 22916 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:37:13.341809 22930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:13.342010 22916 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:37:13.342090 22916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-1/wal
uuid: "4b3cea17d1d14678b3303b28ebe584e9"
format_stamp: "Formatted at 2026-06-20 11:37:13 on dist-test-slave-ctbw"
server_key: "58a018a326a0cf22beffd6aa26026292"
server_key_iv: "b7d68ccb36e26141d0597799821433d0"
server_key_version: "encryptionkey@0"
I20260620 11:37:13.342142 22916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:13.358232 22916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:13.358489 22916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:13.358587 22916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:13.358762 22916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:37:13.359138 22916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:37:13.359174 22916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:13.359203 22916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:37:13.359216 22916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:13.364813 22916 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.130:42741
I20260620 11:37:13.364902 23043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.130:42741 every 8 connection(s)
I20260620 11:37:13.365228 22916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-1/data/info.pb
I20260620 11:37:13.369258 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 22916
I20260620 11:37:13.369366 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-1/wal/instance
I20260620 11:37:13.369575 3022 external_mini_cluster.cc:1428] Setting key 728a32890c8ae50894d5fc800c2848b8
I20260620 11:37:13.370771 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:13.365980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:58185 (local address 127.2.243.190:46251)
0620 11:37:13.366030 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:13.366032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:13.366305 (+ 273us) server_negotiation.cc:407] Connection header received
0620 11:37:13.366833 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:13.366834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:13.366849 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:13.366936 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:13.367636 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.368129 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.369104 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.369430 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.369519 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:13.370472 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:13.370473 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:13.370473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:13.370478 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:13.370495 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:13.370500 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:13.370560 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:13.370653 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 11:37:13.370682 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:37:13.370838 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.2.243.131:0
--local_ip_for_outbound_sockets=127.2.243.131
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:46251
--builtin_ntp_servers=127.2.243.148:46295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 11:37:13.370946 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:13.365950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:46251 (local address 127.2.243.130:58185)
0620 11:37:13.366123 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:13.366185 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:13.366683 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:13.367020 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:13.367024 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:13.367034 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:13.367539 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:13.367544 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.368219 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.368219 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:13.368992 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:13.368994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.369531 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.369532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:13.369643 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:13.370303 (+ 660us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:37:13.370303 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:13.370344 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:37:13.370374 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:13.370572 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:13.370576 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:13.370578 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:37:13.370634 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 11:37:13.370713 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260620 11:37:13.372392 23044 heartbeater.cc:347] Connected to a master server at 127.2.243.190:46251
I20260620 11:37:13.372479 23044 heartbeater.cc:464] Registering TS with master...
I20260620 11:37:13.372678 23044 heartbeater.cc:511] Master 127.2.243.190:46251 requested a full tablet report, sending...
I20260620 11:37:13.373114 22725 ts_manager.cc:194] Registered new tserver with Master: 4b3cea17d1d14678b3303b28ebe584e9 (127.2.243.130:42741)
I20260620 11:37:13.373497 22725 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.243.130:58185
W20260620 11:37:13.450652 23047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:13.450904 23047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:13.450941 23047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:13.452070 23047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:13.452139 23047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:13.452183 23047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.2.243.131
I20260620 11:37:13.453568 23047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:46295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.2.243.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.2.243.131
--webserver_port=0
--tserver_master_addrs=127.2.243.190:46251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.2.243.131
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:13.453832 23047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:13.454123 23047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:13.459153 23052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:13.459160 23053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:13.459205 23055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:13.459231 23047 server_base.cc:1034] running on GCE node
I20260620 11:37:13.459581 23047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:13.459854 23047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:13.461002 23047 hybrid_clock.cc:648] HybridClock initialized: now 1781955433460979 us; error 42 us; skew 500 ppm
I20260620 11:37:13.462385 23047 webserver.cc:460] Webserver started at http://127.2.243.131:38749/ using document root <none> and password file <none>
I20260620 11:37:13.462620 23047 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:13.462666 23047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:13.462810 23047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:13.463907 23047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-2/data/instance:
uuid: "cfec8308ed534404b482c9850b1f5759"
format_stamp: "Formatted at 2026-06-20 11:37:13 on dist-test-slave-ctbw"
server_key: "0966c6ce11c1a0a5a6e05f956ce8c618"
server_key_iv: "75c76c289155be347f213f7e27cc97c9"
server_key_version: "encryptionkey@0"
I20260620 11:37:13.464268 23047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance:
uuid: "cfec8308ed534404b482c9850b1f5759"
format_stamp: "Formatted at 2026-06-20 11:37:13 on dist-test-slave-ctbw"
server_key: "0966c6ce11c1a0a5a6e05f956ce8c618"
server_key_iv: "75c76c289155be347f213f7e27cc97c9"
server_key_version: "encryptionkey@0"
I20260620 11:37:13.465615 23047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:37:13.466375 23061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:13.466550 23047 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:37:13.466650 23047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-2/wal
uuid: "cfec8308ed534404b482c9850b1f5759"
format_stamp: "Formatted at 2026-06-20 11:37:13 on dist-test-slave-ctbw"
server_key: "0966c6ce11c1a0a5a6e05f956ce8c618"
server_key_iv: "75c76c289155be347f213f7e27cc97c9"
server_key_version: "encryptionkey@0"
I20260620 11:37:13.466737 23047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:13.476569 23047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:13.476857 23047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:13.477011 23047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:13.477238 23047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:37:13.477561 23047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:37:13.477622 23047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:13.477654 23047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:37:13.477681 23047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:13.483665 23047 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.131:44697
I20260620 11:37:13.483716 23174 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.131:44697 every 8 connection(s)
I20260620 11:37:13.484014 23047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-2/data/info.pb
I20260620 11:37:13.485719 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 23047
I20260620 11:37:13.485839 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-2/wal/instance
I20260620 11:37:13.486076 3022 external_mini_cluster.cc:1428] Setting key 234cece43beb8a8f8cca75bf46c2ec32
I20260620 11:37:13.489280 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:13.484754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.131:46919 (local address 127.2.243.190:46251)
0620 11:37:13.484836 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:13.484839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:13.485038 (+ 199us) server_negotiation.cc:407] Connection header received
0620 11:37:13.485491 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:13.485492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:13.485502 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:13.485546 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:13.486424 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.487189 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.488121 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.488306 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.488356 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:13.489069 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:13.489070 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:13.489070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:13.489074 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:13.489087 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:13.489091 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:13.489136 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:13.489202 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 11:37:13.489225 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 11:37:13.489464 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:13.484718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:46251 (local address 127.2.243.131:46919)
0620 11:37:13.484918 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:13.484952 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:13.485410 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:13.485622 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:13.485626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:13.485638 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:13.486336 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:13.486341 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.487272 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.487272 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:13.488045 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:13.488047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.488384 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.488385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:13.488485 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:13.488945 (+ 460us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:37:13.488946 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:13.488972 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:37:13.488992 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:13.489138 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:13.489141 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:13.489143 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:37:13.489188 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 11:37:13.489267 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":96,"threads_started":1}
I20260620 11:37:13.489871 23175 heartbeater.cc:347] Connected to a master server at 127.2.243.190:46251
I20260620 11:37:13.489956 23175 heartbeater.cc:464] Registering TS with master...
I20260620 11:37:13.490089 23175 heartbeater.cc:511] Master 127.2.243.190:46251 requested a full tablet report, sending...
I20260620 11:37:13.490401 22725 ts_manager.cc:194] Registered new tserver with Master: cfec8308ed534404b482c9850b1f5759 (127.2.243.131:44697)
I20260620 11:37:13.490716 22725 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.2.243.131:46919
I20260620 11:37:13.498271 3022 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:37:13.502454 22773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:13.499682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41110 (local address 127.2.243.190:46251)
0620 11:37:13.499775 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:13.499777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:13.499870 (+ 93us) server_negotiation.cc:407] Connection header received
0620 11:37:13.499892 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:13.499893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:13.499902 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:13.499950 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:13.500096 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.500611 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.501334 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.501504 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.501553 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:13.502169 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:13.502170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:13.502170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:13.502174 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:13.502186 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:13.502189 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:13.502253 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:13.502319 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 11:37:13.502339 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 11:37:13.503957 22725 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:37:13.505383 22725 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:37:13.511504 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:13.507513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.130:42741 (local address 127.0.0.1:49176)
0620 11:37:13.507674 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:13.507714 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:13.507777 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:13.508122 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:13.508126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:13.508128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:13.508189 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:13.508190 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.509183 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.509184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:13.510290 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:13.510292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.511222 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.511223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:13.511298 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:13.511310 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 11:37:13.511384 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 11:37:13.511432 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":98,"threads_started":1}
I20260620 11:37:13.511519 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:13.507744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.2.243.130:42741)
0620 11:37:13.507922 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:13.507926 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:13.507933 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:37:13.507963 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:13.507988 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:13.508005 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:13.508093 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:13.508308 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.509088 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.510383 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.511111 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.511221 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:13.511323 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:13.511401 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 11:37:13.511435 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":75,"threads_started":1}
I20260620 11:37:13.512009 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:13.507704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:44697 (local address 127.0.0.1:46924)
0620 11:37:13.508022 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:13.508030 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:13.508100 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:13.508360 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:13.508361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:13.508363 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:13.508401 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:13.508403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.509676 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.509677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:13.510800 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:13.510803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.511753 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.511754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:13.511812 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:13.511816 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:37:13.511872 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 11:37:13.511918 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":31,"thread_start_us":65,"threads_started":1}
I20260620 11:37:13.512009 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:13.507484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:44101 (local address 127.0.0.1:43696)
0620 11:37:13.507666 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:13.507706 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:13.507780 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:13.508433 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:13.508434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:13.508435 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:13.508468 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:13.508470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.509701 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.509702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:13.510603 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:13.510605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.511637 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.511639 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:13.511826 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:13.511829 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:37:13.511872 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 11:37:13.511918 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":22,"thread_start_us":72,"threads_started":1}
I20260620 11:37:13.512194 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:13.507578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43696 (local address 127.2.243.129:44101)
0620 11:37:13.508229 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:13.508233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:13.508245 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:37:13.508284 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:13.508286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:13.508300 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:13.508369 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:13.508542 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.509620 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.510700 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.511447 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.511520 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:13.511601 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:13.512049 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 11:37:13.512099 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":72,"threads_started":1}
I20260620 11:37:13.512450 22978 tablet_service.cc:1467] Processing CreateTablet for tablet 950dd121c8314c228507d2729703dd7e (DEFAULT_TABLE table=test-table [id=0cb0ff6607c3491fab18778bf6c1979e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:37:13.512807 22978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 950dd121c8314c228507d2729703dd7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:13.512895 22847 tablet_service.cc:1467] Processing CreateTablet for tablet 950dd121c8314c228507d2729703dd7e (DEFAULT_TABLE table=test-table [id=0cb0ff6607c3491fab18778bf6c1979e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:37:13.513146 22847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 950dd121c8314c228507d2729703dd7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:13.515062 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:13.507882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46924 (local address 127.2.243.131:44697)
0620 11:37:13.508064 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:13.508067 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:13.508077 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:37:13.508198 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:13.508201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:13.508216 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:13.508295 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:13.508479 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.509571 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.510892 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.511586 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.511646 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:13.511721 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:13.511891 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 11:37:13.511935 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":82,"threads_started":1}
I20260620 11:37:13.515676 23189 tablet_bootstrap.cc:492] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9: Bootstrap starting.
I20260620 11:37:13.515684 23109 tablet_service.cc:1467] Processing CreateTablet for tablet 950dd121c8314c228507d2729703dd7e (DEFAULT_TABLE table=test-table [id=0cb0ff6607c3491fab18778bf6c1979e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:37:13.515820 23190 tablet_bootstrap.cc:492] T 950dd121c8314c228507d2729703dd7e P dfcc4c45e2534128b5fce461b90740b0: Bootstrap starting.
I20260620 11:37:13.516032 23109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 950dd121c8314c228507d2729703dd7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:13.516306 23190 tablet_bootstrap.cc:654] T 950dd121c8314c228507d2729703dd7e P dfcc4c45e2534128b5fce461b90740b0: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:13.516348 23189 tablet_bootstrap.cc:654] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:13.516520 23190 log.cc:824] T 950dd121c8314c228507d2729703dd7e P dfcc4c45e2534128b5fce461b90740b0: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:13.516630 23189 log.cc:824] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:13.517130 23190 tablet_bootstrap.cc:492] T 950dd121c8314c228507d2729703dd7e P dfcc4c45e2534128b5fce461b90740b0: No bootstrap required, opened a new log
I20260620 11:37:13.517220 23190 ts_tablet_manager.cc:1397] T 950dd121c8314c228507d2729703dd7e P dfcc4c45e2534128b5fce461b90740b0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:37:13.517280 23189 tablet_bootstrap.cc:492] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9: No bootstrap required, opened a new log
I20260620 11:37:13.517365 23189 ts_tablet_manager.cc:1397] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:37:13.519395 23190 raft_consensus.cc:348] T 950dd121c8314c228507d2729703dd7e P dfcc4c45e2534128b5fce461b90740b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc4c45e2534128b5fce461b90740b0" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44101 } } peers { permanent_uuid: "4b3cea17d1d14678b3303b28ebe584e9" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42741 } } peers { permanent_uuid: "cfec8308ed534404b482c9850b1f5759" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 44697 } }
I20260620 11:37:13.519573 23190 raft_consensus.cc:374] T 950dd121c8314c228507d2729703dd7e P dfcc4c45e2534128b5fce461b90740b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:13.519618 23190 raft_consensus.cc:729] T 950dd121c8314c228507d2729703dd7e P dfcc4c45e2534128b5fce461b90740b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcc4c45e2534128b5fce461b90740b0, State: Initialized, Role: FOLLOWER
I20260620 11:37:13.519542 23193 tablet_bootstrap.cc:492] T 950dd121c8314c228507d2729703dd7e P cfec8308ed534404b482c9850b1f5759: Bootstrap starting.
I20260620 11:37:13.519761 23190 consensus_queue.cc:260] T 950dd121c8314c228507d2729703dd7e P dfcc4c45e2534128b5fce461b90740b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc4c45e2534128b5fce461b90740b0" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44101 } } peers { permanent_uuid: "4b3cea17d1d14678b3303b28ebe584e9" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42741 } } peers { permanent_uuid: "cfec8308ed534404b482c9850b1f5759" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 44697 } }
I20260620 11:37:13.520036 23190 ts_tablet_manager.cc:1428] T 950dd121c8314c228507d2729703dd7e P dfcc4c45e2534128b5fce461b90740b0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:37:13.520035 23189 raft_consensus.cc:348] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc4c45e2534128b5fce461b90740b0" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44101 } } peers { permanent_uuid: "4b3cea17d1d14678b3303b28ebe584e9" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42741 } } peers { permanent_uuid: "cfec8308ed534404b482c9850b1f5759" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 44697 } }
I20260620 11:37:13.520182 22913 heartbeater.cc:503] Master 127.2.243.190:46251 was elected leader, sending a full tablet report...
I20260620 11:37:13.520195 23189 raft_consensus.cc:374] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:13.520248 23189 raft_consensus.cc:729] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b3cea17d1d14678b3303b28ebe584e9, State: Initialized, Role: FOLLOWER
I20260620 11:37:13.520366 23193 tablet_bootstrap.cc:654] T 950dd121c8314c228507d2729703dd7e P cfec8308ed534404b482c9850b1f5759: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:13.520357 23189 consensus_queue.cc:260] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc4c45e2534128b5fce461b90740b0" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44101 } } peers { permanent_uuid: "4b3cea17d1d14678b3303b28ebe584e9" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42741 } } peers { permanent_uuid: "cfec8308ed534404b482c9850b1f5759" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 44697 } }
I20260620 11:37:13.520601 23189 ts_tablet_manager.cc:1428] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:37:13.520661 23044 heartbeater.cc:503] Master 127.2.243.190:46251 was elected leader, sending a full tablet report...
I20260620 11:37:13.520678 23193 log.cc:824] T 950dd121c8314c228507d2729703dd7e P cfec8308ed534404b482c9850b1f5759: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:13.521523 23193 tablet_bootstrap.cc:492] T 950dd121c8314c228507d2729703dd7e P cfec8308ed534404b482c9850b1f5759: No bootstrap required, opened a new log
I20260620 11:37:13.521620 23193 ts_tablet_manager.cc:1397] T 950dd121c8314c228507d2729703dd7e P cfec8308ed534404b482c9850b1f5759: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:37:13.523586 23193 raft_consensus.cc:348] T 950dd121c8314c228507d2729703dd7e P cfec8308ed534404b482c9850b1f5759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc4c45e2534128b5fce461b90740b0" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44101 } } peers { permanent_uuid: "4b3cea17d1d14678b3303b28ebe584e9" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42741 } } peers { permanent_uuid: "cfec8308ed534404b482c9850b1f5759" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 44697 } }
I20260620 11:37:13.523705 23193 raft_consensus.cc:374] T 950dd121c8314c228507d2729703dd7e P cfec8308ed534404b482c9850b1f5759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:13.523728 23193 raft_consensus.cc:729] T 950dd121c8314c228507d2729703dd7e P cfec8308ed534404b482c9850b1f5759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfec8308ed534404b482c9850b1f5759, State: Initialized, Role: FOLLOWER
I20260620 11:37:13.523841 23193 consensus_queue.cc:260] T 950dd121c8314c228507d2729703dd7e P cfec8308ed534404b482c9850b1f5759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc4c45e2534128b5fce461b90740b0" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44101 } } peers { permanent_uuid: "4b3cea17d1d14678b3303b28ebe584e9" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42741 } } peers { permanent_uuid: "cfec8308ed534404b482c9850b1f5759" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 44697 } }
I20260620 11:37:13.524052 23193 ts_tablet_manager.cc:1428] T 950dd121c8314c228507d2729703dd7e P cfec8308ed534404b482c9850b1f5759: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:37:13.524147 23175 heartbeater.cc:503] Master 127.2.243.190:46251 was elected leader, sending a full tablet report...
I20260620 11:37:13.577943 23195 raft_consensus.cc:482] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:37:13.578060 23195 raft_consensus.cc:504] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc4c45e2534128b5fce461b90740b0" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44101 } } peers { permanent_uuid: "4b3cea17d1d14678b3303b28ebe584e9" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42741 } } peers { permanent_uuid: "cfec8308ed534404b482c9850b1f5759" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 44697 } }
I20260620 11:37:13.578465 23195 leader_election.cc:288] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dfcc4c45e2534128b5fce461b90740b0 (127.2.243.129:44101), cfec8308ed534404b482c9850b1f5759 (127.2.243.131:44697)
I20260620 11:37:13.581722 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:13.579008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:41363 (local address 127.2.243.129:44101)
0620 11:37:13.579056 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:13.579058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:13.579073 (+ 15us) server_negotiation.cc:407] Connection header received
0620 11:37:13.579097 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:13.579098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:13.579109 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:13.579211 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:13.579372 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.579848 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.580836 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.581348 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.581437 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:13.581446 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:13.581614 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 11:37:13.581652 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:37:13.581722 23046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:13.578727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:44101 (local address 127.2.243.130:41363)
0620 11:37:13.578782 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:13.578795 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:13.578869 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:13.579220 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:13.579221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:13.579223 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:13.579295 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:13.579297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.579933 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.579934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:13.580738 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:13.580740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.581453 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.581455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:13.581542 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:13.581547 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:37:13.581605 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 11:37:13.581646 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 11:37:13.581876 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:13.578965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.2.243.130:58481 (local address 127.2.243.131:44697)
0620 11:37:13.579029 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:13.579031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:13.579159 (+ 128us) server_negotiation.cc:407] Connection header received
0620 11:37:13.579228 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:13.579229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:13.579237 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:13.579306 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:37:13.579461 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.579923 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.580948 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.581500 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.581594 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:13.581601 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:13.581780 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 11:37:13.581814 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 11:37:13.581887 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:13.578838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.131:44697 (local address 127.2.243.130:58481)
0620 11:37:13.579091 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:13.579099 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:13.579172 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:13.579315 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:13.579316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:13.579317 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:37:13.579389 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:13.579391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.580005 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.580006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:13.580859 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:13.580861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.581605 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.581606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:13.581687 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:13.581690 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:37:13.581772 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 11:37:13.581809 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":151,"threads_started":1}
I20260620 11:37:13.582544 23129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950dd121c8314c228507d2729703dd7e" candidate_uuid: "4b3cea17d1d14678b3303b28ebe584e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfec8308ed534404b482c9850b1f5759" is_pre_election: true
I20260620 11:37:13.582569 22867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950dd121c8314c228507d2729703dd7e" candidate_uuid: "4b3cea17d1d14678b3303b28ebe584e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcc4c45e2534128b5fce461b90740b0" is_pre_election: true
I20260620 11:37:13.582713 23129 raft_consensus.cc:2454] T 950dd121c8314c228507d2729703dd7e P cfec8308ed534404b482c9850b1f5759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b3cea17d1d14678b3303b28ebe584e9 in term 0.
I20260620 11:37:13.582713 22867 raft_consensus.cc:2454] T 950dd121c8314c228507d2729703dd7e P dfcc4c45e2534128b5fce461b90740b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b3cea17d1d14678b3303b28ebe584e9 in term 0.
I20260620 11:37:13.582993 22933 leader_election.cc:302] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b3cea17d1d14678b3303b28ebe584e9, dfcc4c45e2534128b5fce461b90740b0; no voters:
I20260620 11:37:13.583110 23195 raft_consensus.cc:2781] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:37:13.583163 23195 raft_consensus.cc:482] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:37:13.583191 23195 raft_consensus.cc:3037] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:13.583899 23195 raft_consensus.cc:504] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc4c45e2534128b5fce461b90740b0" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44101 } } peers { permanent_uuid: "4b3cea17d1d14678b3303b28ebe584e9" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42741 } } peers { permanent_uuid: "cfec8308ed534404b482c9850b1f5759" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 44697 } }
I20260620 11:37:13.584039 23195 leader_election.cc:288] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9 [CANDIDATE]: Term 1 election: Requested vote from peers dfcc4c45e2534128b5fce461b90740b0 (127.2.243.129:44101), cfec8308ed534404b482c9850b1f5759 (127.2.243.131:44697)
I20260620 11:37:13.584264 22867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950dd121c8314c228507d2729703dd7e" candidate_uuid: "4b3cea17d1d14678b3303b28ebe584e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfcc4c45e2534128b5fce461b90740b0"
I20260620 11:37:13.584285 23129 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950dd121c8314c228507d2729703dd7e" candidate_uuid: "4b3cea17d1d14678b3303b28ebe584e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfec8308ed534404b482c9850b1f5759"
I20260620 11:37:13.584352 23129 raft_consensus.cc:3037] T 950dd121c8314c228507d2729703dd7e P cfec8308ed534404b482c9850b1f5759 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:13.584352 22867 raft_consensus.cc:3037] T 950dd121c8314c228507d2729703dd7e P dfcc4c45e2534128b5fce461b90740b0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:13.585299 23129 raft_consensus.cc:2454] T 950dd121c8314c228507d2729703dd7e P cfec8308ed534404b482c9850b1f5759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b3cea17d1d14678b3303b28ebe584e9 in term 1.
I20260620 11:37:13.585299 22867 raft_consensus.cc:2454] T 950dd121c8314c228507d2729703dd7e P dfcc4c45e2534128b5fce461b90740b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b3cea17d1d14678b3303b28ebe584e9 in term 1.
I20260620 11:37:13.585471 22933 leader_election.cc:302] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b3cea17d1d14678b3303b28ebe584e9, dfcc4c45e2534128b5fce461b90740b0; no voters:
I20260620 11:37:13.585588 23195 raft_consensus.cc:2781] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:13.585737 23195 raft_consensus.cc:686] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9 [term 1 LEADER]: Becoming Leader. State: Replica: 4b3cea17d1d14678b3303b28ebe584e9, State: Running, Role: LEADER
I20260620 11:37:13.585840 23195 consensus_queue.cc:237] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc4c45e2534128b5fce461b90740b0" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44101 } } peers { permanent_uuid: "4b3cea17d1d14678b3303b28ebe584e9" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42741 } } peers { permanent_uuid: "cfec8308ed534404b482c9850b1f5759" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 44697 } }
I20260620 11:37:13.586674 22725 catalog_manager.cc:5495] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b3cea17d1d14678b3303b28ebe584e9 (127.2.243.130). New cstate: current_term: 1 leader_uuid: "4b3cea17d1d14678b3303b28ebe584e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dfcc4c45e2534128b5fce461b90740b0" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b3cea17d1d14678b3303b28ebe584e9" member_type: VOTER last_known_addr { host: "127.2.243.130" port: 42741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfec8308ed534404b482c9850b1f5759" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 44697 } health_report { overall_health: UNKNOWN } } }
I20260620 11:37:13.605935 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:13.603492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49178 (local address 127.2.243.130:42741)
0620 11:37:13.603554 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:13.603556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:13.603563 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:37:13.603597 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:13.603598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:13.603614 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:13.603718 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:13.603859 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.604321 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.605162 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:13.605347 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:13.605403 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:13.605592 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:13.605694 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:13.605769 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:13.605823 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 11:37:13.605856 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 11:37:13.608004 23129 raft_consensus.cc:1261] T 950dd121c8314c228507d2729703dd7e P cfec8308ed534404b482c9850b1f5759 [term 1 FOLLOWER]: Refusing update from remote peer 4b3cea17d1d14678b3303b28ebe584e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:37:13.608060 22867 raft_consensus.cc:1261] T 950dd121c8314c228507d2729703dd7e P dfcc4c45e2534128b5fce461b90740b0 [term 1 FOLLOWER]: Refusing update from remote peer 4b3cea17d1d14678b3303b28ebe584e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:37:13.608392 23195 consensus_queue.cc:1035] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfec8308ed534404b482c9850b1f5759" member_type: VOTER last_known_addr { host: "127.2.243.131" port: 44697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:37:13.608549 23199 consensus_queue.cc:1035] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfcc4c45e2534128b5fce461b90740b0" member_type: VOTER last_known_addr { host: "127.2.243.129" port: 44101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:37:13.610947 23202 mvcc.cc:205] Tried to move back new op lower bound from 7298889456055623680 to 7298889455968464896. Current Snapshot: MvccSnapshot[applied={T|T < 7298889456055623680}]
I20260620 11:37:13.611187 23203 mvcc.cc:205] Tried to move back new op lower bound from 7298889456055623680 to 7298889455968464896. Current Snapshot: MvccSnapshot[applied={T|T < 7298889456055623680}]
I20260620 11:37:13.611462 23204 mvcc.cc:205] Tried to move back new op lower bound from 7298889456055623680 to 7298889455968464896. Current Snapshot: MvccSnapshot[applied={T|T < 7298889456055623680}]
I20260620 11:37:13.613399 22725 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:37:13.613504 22725 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:37:13.614260 22725 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f36cf3a2e1d24b0a93fd2e0dde346cdd: Sending DeleteTablet for 3 replicas of tablet 950dd121c8314c228507d2729703dd7e
I20260620 11:37:13.614558 22847 tablet_service.cc:1514] Processing DeleteTablet for tablet 950dd121c8314c228507d2729703dd7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:37:13 UTC) from {username='slave'} at 127.0.0.1:43696
I20260620 11:37:13.614593 22978 tablet_service.cc:1514] Processing DeleteTablet for tablet 950dd121c8314c228507d2729703dd7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:37:13 UTC) from {username='slave'} at 127.0.0.1:49176
I20260620 11:37:13.614610 23109 tablet_service.cc:1514] Processing DeleteTablet for tablet 950dd121c8314c228507d2729703dd7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:37:13 UTC) from {username='slave'} at 127.0.0.1:46924
I20260620 11:37:13.614921 23213 tablet_replica.cc:330] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9: stopping tablet replica
I20260620 11:37:13.615025 23213 raft_consensus.cc:2229] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:37:13.615116 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 22785
I20260620 11:37:13.615144 23213 raft_consensus.cc:2258] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:37:13.616006 23214 tablet_replica.cc:330] T 950dd121c8314c228507d2729703dd7e P cfec8308ed534404b482c9850b1f5759: stopping tablet replica
I20260620 11:37:13.616097 23214 raft_consensus.cc:2229] T 950dd121c8314c228507d2729703dd7e P cfec8308ed534404b482c9850b1f5759 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:37:13.616205 23214 raft_consensus.cc:2258] T 950dd121c8314c228507d2729703dd7e P cfec8308ed534404b482c9850b1f5759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:37:13.616284 23213 ts_tablet_manager.cc:1905] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:37:13.616631 23214 ts_tablet_manager.cc:1905] T 950dd121c8314c228507d2729703dd7e P cfec8308ed534404b482c9850b1f5759: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:37:13.618073 23213 ts_tablet_manager.cc:1918] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:37:13.618188 23213 log.cc:1196] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-1/wal/wals/950dd121c8314c228507d2729703dd7e
I20260620 11:37:13.618211 23214 ts_tablet_manager.cc:1918] T 950dd121c8314c228507d2729703dd7e P cfec8308ed534404b482c9850b1f5759: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:37:13.618274 23214 log.cc:1196] T 950dd121c8314c228507d2729703dd7e P cfec8308ed534404b482c9850b1f5759: Deleting WAL directory at /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955192827595-3022-0/minicluster-data/ts-2/wal/wals/950dd121c8314c228507d2729703dd7e
I20260620 11:37:13.618464 23213 ts_tablet_manager.cc:1939] T 950dd121c8314c228507d2729703dd7e P 4b3cea17d1d14678b3303b28ebe584e9: Deleting consensus metadata
I20260620 11:37:13.618511 23214 ts_tablet_manager.cc:1939] T 950dd121c8314c228507d2729703dd7e P cfec8308ed534404b482c9850b1f5759: Deleting consensus metadata
I20260620 11:37:13.619144 22709 catalog_manager.cc:4841] TS 4b3cea17d1d14678b3303b28ebe584e9 (127.2.243.130:42741): tablet 950dd121c8314c228507d2729703dd7e (table test-table [id=0cb0ff6607c3491fab18778bf6c1979e]) successfully deleted
I20260620 11:37:13.619331 22711 catalog_manager.cc:4841] TS cfec8308ed534404b482c9850b1f5759 (127.2.243.131:44697): tablet 950dd121c8314c228507d2729703dd7e (table test-table [id=0cb0ff6607c3491fab18778bf6c1979e]) successfully deleted
W20260620 11:37:13.620724 22711 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.2.243.129:44101 (error 108)
I20260620 11:37:13.620817 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 22916
I20260620 11:37:13.621730 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:13.621573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.129:44101 (local address 127.0.0.1:43710)
0620 11:37:13.621644 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:13.621691 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.129:44101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 11:37:13.621881 22711 catalog_manager.cc:4584] TS dfcc4c45e2534128b5fce461b90740b0 (127.2.243.129:44101): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 950dd121c8314c228507d2729703dd7e: Network error: Client connection negotiation failed: client connection to 127.2.243.129:44101: connect: Connection refused (error 111)
I20260620 11:37:13.626346 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 23047
I20260620 11:37:13.630776 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 22694
2026-06-20T11:37:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (731 ms)
[----------] 6 tests from EncryptionPolicyTest (8318 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:13 dist-test-slave-ctbw krb5kdc[23218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:13 dist-test-slave-ctbw krb5kdc[23218](info): set up 2 sockets
Jun 20 11:37:13 dist-test-slave-ctbw krb5kdc[23218](info): commencing operation
krb5kdc: starting...
W20260620 11:37:14.675135 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:14 dist-test-slave-ctbw krb5kdc[23218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:37:14.792084 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44989
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:14.870926 23231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:14.871309 23231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:14.871361 23231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:14.872515 23231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:14.872555 23231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:14.872570 23231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:14.872582 23231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:14.872630 23231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:14.874078 23231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44989
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:14.874334 23231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:14.874571 23231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 11:37:14.879485 23231 server_base.cc:1034] running on GCE node
W20260620 11:37:14.879434 23236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:14.879460 23237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:14.879451 23239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:14.879988 23231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:14.880296 23231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:14.881439 23231 hybrid_clock.cc:648] HybridClock initialized: now 1781955434881416 us; error 35 us; skew 500 ppm
Jun 20 11:37:14 dist-test-slave-ctbw krb5kdc[23218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:14.883961 23231 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:37:14.884428 23231 webserver.cc:460] Webserver started at http://127.0.0.1:35017/ using document root <none> and password file <none>
I20260620 11:37:14.884615 23231 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:14.884655 23231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:14.884740 23231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:14.885704 23231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "b3468a6a260040a1a7bfb0bab1bfa14d"
format_stamp: "Formatted at 2026-06-20 11:37:14 on dist-test-slave-ctbw"
server_key: "22a51deaed239d96c14e3f49ba5a2684"
server_key_iv: "91290b693e2b21e0ba81eeae008cda0a"
server_key_version: "encryptionkey@0"
I20260620 11:37:14.885988 23231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "b3468a6a260040a1a7bfb0bab1bfa14d"
format_stamp: "Formatted at 2026-06-20 11:37:14 on dist-test-slave-ctbw"
server_key: "22a51deaed239d96c14e3f49ba5a2684"
server_key_iv: "91290b693e2b21e0ba81eeae008cda0a"
server_key_version: "encryptionkey@0"
I20260620 11:37:14.887372 23231 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:37:14.888084 23246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:14.888259 23231 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:37:14.888365 23231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "b3468a6a260040a1a7bfb0bab1bfa14d"
format_stamp: "Formatted at 2026-06-20 11:37:14 on dist-test-slave-ctbw"
server_key: "22a51deaed239d96c14e3f49ba5a2684"
server_key_iv: "91290b693e2b21e0ba81eeae008cda0a"
server_key_version: "encryptionkey@0"
I20260620 11:37:14.888449 23231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:14.905911 23231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:14.907819 23231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:14.907987 23231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:14.912009 23231 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44989
I20260620 11:37:14.912042 23298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44989 every 8 connection(s)
I20260620 11:37:14.912384 23231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:14.913367 23299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:14.915620 23299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d: Bootstrap starting.
I20260620 11:37:14.916146 23299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:14.916368 23299 log.cc:824] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:14.916749 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 23231
I20260620 11:37:14.916883 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:14.917088 3022 external_mini_cluster.cc:1428] Setting key 088f37c0c709b7bceb64156390700cae
I20260620 11:37:14.917191 23299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d: No bootstrap required, opened a new log
I20260620 11:37:14.919163 23299 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3468a6a260040a1a7bfb0bab1bfa14d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } }
I20260620 11:37:14.919322 23299 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:14.919370 23299 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3468a6a260040a1a7bfb0bab1bfa14d, State: Initialized, Role: FOLLOWER
I20260620 11:37:14.919502 23299 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3468a6a260040a1a7bfb0bab1bfa14d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } }
I20260620 11:37:14.919584 23299 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:14.919608 23299 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:14.919670 23299 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:14.920228 23299 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3468a6a260040a1a7bfb0bab1bfa14d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } }
I20260620 11:37:14.920310 23299 leader_election.cc:302] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3468a6a260040a1a7bfb0bab1bfa14d; no voters:
I20260620 11:37:14.920492 23299 leader_election.cc:288] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:14.920532 23304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:14.920686 23304 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d [term 1 LEADER]: Becoming Leader. State: Replica: b3468a6a260040a1a7bfb0bab1bfa14d, State: Running, Role: LEADER
I20260620 11:37:14.920770 23299 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:14.920868 23304 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3468a6a260040a1a7bfb0bab1bfa14d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } }
I20260620 11:37:14.922173 23305 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3468a6a260040a1a7bfb0bab1bfa14d. Latest consensus state: current_term: 1 leader_uuid: "b3468a6a260040a1a7bfb0bab1bfa14d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3468a6a260040a1a7bfb0bab1bfa14d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } } }
I20260620 11:37:14.922879 23305 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:14.922236 23306 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3468a6a260040a1a7bfb0bab1bfa14d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3468a6a260040a1a7bfb0bab1bfa14d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44989 } } }
I20260620 11:37:14.923249 23306 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d [sys.catalog]: This master's current role is: LEADER
W20260620 11:37:14.922587 23319 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:37:14.923341 23319 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:37:14.924346 23321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:14.924536 23321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 11:37:14 dist-test-slave-ctbw krb5kdc[23218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 11:37:14.927596 23321 catalog_manager.cc:1347] Generated new cluster ID: d545a38686454fd7914497cfdbc17e92
I20260620 11:37:14.927649 23321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:37:14.929085 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:14.917631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53143 (local address 127.0.0.1:44989)
0620 11:37:14.917845 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:14.917855 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:14.917890 (+ 35us) server_negotiation.cc:407] Connection header received
0620 11:37:14.918435 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:14.918441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:14.918492 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:14.918650 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:14.919355 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:14.920832 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:14.921883 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:14.922496 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:14.922608 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:14.925517 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:14.925521 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:14.925521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:14.925532 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:14.927440 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:14.927902 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:14.927905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:14.927906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:14.927997 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:14.928237 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:14.928238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:14.928238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:14.928473 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:14.928654 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:14.928787 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 11:37:14.928889 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":89,"threads_started":1}
I20260620 11:37:14.930084 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:37:14.938336 23321 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:14.939036 23321 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:14.941377 23303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:14.931446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40943 (local address 127.0.0.1:44989)
0620 11:37:14.931486 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:14.931488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:14.935047 (+ 3559us) server_negotiation.cc:407] Connection header received
0620 11:37:14.935111 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:14.935112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:14.935129 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:14.935177 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:14.935690 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:14.936200 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:14.936969 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:14.937200 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:14.937291 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:14.938693 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:14.938695 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:14.938695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:14.938702 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:14.940187 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:14.940565 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:14.940566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:14.940566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:14.940624 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:14.940856 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:14.940856 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:14.940857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:14.940964 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:14.941080 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:14.941203 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 11:37:14.941265 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:37:14.944312 23321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3468a6a260040a1a7bfb0bab1bfa14d: Generated new TSK 0
I20260620 11:37:14.944475 23321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:37:14.950263 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 23231
W20260620 11:37:14.954550 23216 connection.cc:535] client connection to 127.0.0.1:44989 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44989): Connection reset by peer (error 104)
2026-06-20T11:37:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:14 dist-test-slave-ctbw krb5kdc[23330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:14 dist-test-slave-ctbw krb5kdc[23330](info): set up 2 sockets
Jun 20 11:37:14 dist-test-slave-ctbw krb5kdc[23330](info): commencing operation
krb5kdc: starting...
W20260620 11:37:15.987157 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:16 dist-test-slave-ctbw krb5kdc[23330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:37:16.101248 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44733
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:16.180891 23343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:16.181116 23343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:16.181150 23343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:16.182299 23343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:16.182370 23343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:16.182395 23343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:16.182423 23343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:16.182452 23343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:16.183934 23343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44733
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:16.184233 23343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:16.184463 23343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:16.189131 23349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:16.189128 23348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:16.189158 23351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:16.189361 23343 server_base.cc:1034] running on GCE node
I20260620 11:37:16.189597 23343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:16.189901 23343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:16.191092 23343 hybrid_clock.cc:648] HybridClock initialized: now 1781955436191076 us; error 27 us; skew 500 ppm
Jun 20 11:37:16 dist-test-slave-ctbw krb5kdc[23330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955436, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:16.193449 23343 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:37:16.193847 23343 webserver.cc:460] Webserver started at http://127.0.0.1:33193/ using document root <none> and password file <none>
I20260620 11:37:16.194087 23343 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:16.194155 23343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:16.194298 23343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:16.195416 23343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "80edcc66804544cfa5d8957eca10744e"
format_stamp: "Formatted at 2026-06-20 11:37:16 on dist-test-slave-ctbw"
server_key: "b8028368c894a1fd8d495aaf239990cd"
server_key_iv: "d86f725b6e73c05d4f2659d011333185"
server_key_version: "encryptionkey@0"
I20260620 11:37:16.195799 23343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "80edcc66804544cfa5d8957eca10744e"
format_stamp: "Formatted at 2026-06-20 11:37:16 on dist-test-slave-ctbw"
server_key: "b8028368c894a1fd8d495aaf239990cd"
server_key_iv: "d86f725b6e73c05d4f2659d011333185"
server_key_version: "encryptionkey@0"
I20260620 11:37:16.197024 23343 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:37:16.197746 23358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:16.197908 23343 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:37:16.198007 23343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "80edcc66804544cfa5d8957eca10744e"
format_stamp: "Formatted at 2026-06-20 11:37:16 on dist-test-slave-ctbw"
server_key: "b8028368c894a1fd8d495aaf239990cd"
server_key_iv: "d86f725b6e73c05d4f2659d011333185"
server_key_version: "encryptionkey@0"
I20260620 11:37:16.198093 23343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:16.204926 23343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:16.207237 23343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:16.207391 23343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:16.211340 23343 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44733
I20260620 11:37:16.211410 23410 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44733 every 8 connection(s)
I20260620 11:37:16.211760 23343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:16.212716 23411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:16.214745 23411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e: Bootstrap starting.
I20260620 11:37:16.215301 23411 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:16.215613 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 23343
I20260620 11:37:16.215619 23411 log.cc:824] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:16.215734 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:16.215941 3022 external_mini_cluster.cc:1428] Setting key 9228a942e2be8bd7a763708509b3bae7
I20260620 11:37:16.216351 23411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e: No bootstrap required, opened a new log
I20260620 11:37:16.218015 23411 raft_consensus.cc:348] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80edcc66804544cfa5d8957eca10744e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44733 } }
I20260620 11:37:16.218178 23411 raft_consensus.cc:374] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:16.218259 23411 raft_consensus.cc:729] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80edcc66804544cfa5d8957eca10744e, State: Initialized, Role: FOLLOWER
I20260620 11:37:16.218413 23411 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80edcc66804544cfa5d8957eca10744e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44733 } }
I20260620 11:37:16.218492 23411 raft_consensus.cc:388] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:16.218516 23411 raft_consensus.cc:482] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:16.218585 23411 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:16.219213 23411 raft_consensus.cc:504] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80edcc66804544cfa5d8957eca10744e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44733 } }
I20260620 11:37:16.219296 23411 leader_election.cc:302] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 80edcc66804544cfa5d8957eca10744e; no voters:
I20260620 11:37:16.219470 23411 leader_election.cc:288] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:16.219513 23416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:16.219674 23416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e [term 1 LEADER]: Becoming Leader. State: Replica: 80edcc66804544cfa5d8957eca10744e, State: Running, Role: LEADER
I20260620 11:37:16.219756 23411 sys_catalog.cc:564] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:16.219794 23416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80edcc66804544cfa5d8957eca10744e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44733 } }
I20260620 11:37:16.220161 23417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 80edcc66804544cfa5d8957eca10744e. Latest consensus state: current_term: 1 leader_uuid: "80edcc66804544cfa5d8957eca10744e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80edcc66804544cfa5d8957eca10744e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44733 } } }
I20260620 11:37:16.220260 23417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:16.220499 23422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:16.220963 23418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "80edcc66804544cfa5d8957eca10744e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "80edcc66804544cfa5d8957eca10744e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44733 } } }
I20260620 11:37:16.221076 23418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:16.221176 23422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:16.223800 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:16.216420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42215 (local address 127.0.0.1:44733)
0620 11:37:16.216591 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:16.216600 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:16.216630 (+ 30us) server_negotiation.cc:407] Connection header received
0620 11:37:16.218006 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:16.218012 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:16.218056 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:16.218192 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:16.218436 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:16.219966 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:16.221730 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:16.222140 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:16.222265 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:16.223050 (+ 785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:16.223054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:16.223054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:16.223250 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:16.223288 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:16.223304 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:16.223374 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:16.223559 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 11:37:16.223589 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260620 11:37:16.224864 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:37:16.227234 23422 catalog_manager.cc:1347] Generated new cluster ID: 966fa2c9e046497292a93c5ea999823e
I20260620 11:37:16.227298 23422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:37:16.230743 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:16.226130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55739 (local address 127.0.0.1:44733)
0620 11:37:16.226170 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:16.226171 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:16.227381 (+ 1210us) server_negotiation.cc:407] Connection header received
0620 11:37:16.227485 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:16.227487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:16.227500 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:16.227575 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:16.227716 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:16.228432 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:16.229409 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:16.229726 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:16.229800 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:16.230497 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:16.230499 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:16.230499 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:16.230503 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:16.230519 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:16.230524 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:16.230589 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:16.230662 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 11:37:16.230675 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:37:16.255313 23422 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:16.256114 23422 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:16.263345 23422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 80edcc66804544cfa5d8957eca10744e: Generated new TSK 0
I20260620 11:37:16.263569 23422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:37:16.266649 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 23343
W20260620 11:37:16.271091 23328 connection.cc:535] client connection to 127.0.0.1:44733 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44733): Connection reset by peer (error 104)
2026-06-20T11:37:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1316 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:16 dist-test-slave-ctbw krb5kdc[23442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:16 dist-test-slave-ctbw krb5kdc[23442](info): set up 2 sockets
Jun 20 11:37:16 dist-test-slave-ctbw krb5kdc[23442](info): commencing operation
krb5kdc: starting...
W20260620 11:37:17.307169 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:17 dist-test-slave-ctbw krb5kdc[23442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:37:17.423270 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33801
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:17.502909 23455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:17.503124 23455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:17.503149 23455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:17.504403 23455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:17.504448 23455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:17.504462 23455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:17.504475 23455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:17.504487 23455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:17.506114 23455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33801
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:17.506384 23455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:17.506623 23455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:17.511574 23461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:17.511581 23463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:17.511629 23460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:17.511924 23455 server_base.cc:1034] running on GCE node
I20260620 11:37:17.512163 23455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:17.512460 23455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:17.513657 23455 hybrid_clock.cc:648] HybridClock initialized: now 1781955437513632 us; error 37 us; skew 500 ppm
Jun 20 11:37:17 dist-test-slave-ctbw krb5kdc[23442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:17.519810 23455 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:37:17.520264 23455 webserver.cc:460] Webserver started at http://127.0.0.1:36395/ using document root <none> and password file <none>
I20260620 11:37:17.520527 23455 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:17.520601 23455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:17.520750 23455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:17.521826 23455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "be4168f9d721465e990dd2a2c78d8a39"
format_stamp: "Formatted at 2026-06-20 11:37:17 on dist-test-slave-ctbw"
server_key: "6d672766db08c88c468b9b76e99f9005"
server_key_iv: "488ae24111bc44640f91275689397530"
server_key_version: "encryptionkey@0"
I20260620 11:37:17.522193 23455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "be4168f9d721465e990dd2a2c78d8a39"
format_stamp: "Formatted at 2026-06-20 11:37:17 on dist-test-slave-ctbw"
server_key: "6d672766db08c88c468b9b76e99f9005"
server_key_iv: "488ae24111bc44640f91275689397530"
server_key_version: "encryptionkey@0"
I20260620 11:37:17.523658 23455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:37:17.524485 23470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:17.524719 23455 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:37:17.524842 23455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "be4168f9d721465e990dd2a2c78d8a39"
format_stamp: "Formatted at 2026-06-20 11:37:17 on dist-test-slave-ctbw"
server_key: "6d672766db08c88c468b9b76e99f9005"
server_key_iv: "488ae24111bc44640f91275689397530"
server_key_version: "encryptionkey@0"
I20260620 11:37:17.524921 23455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:17.528442 23455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:17.530490 23455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:17.530644 23455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:17.534528 23455 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33801
I20260620 11:37:17.534588 23522 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33801 every 8 connection(s)
I20260620 11:37:17.534992 23455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:17.535950 23523 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:17.537657 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 23455
I20260620 11:37:17.537797 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:17.537990 3022 external_mini_cluster.cc:1428] Setting key 474d0d4cf122e2a66ca1b15cc3b5ba2f
I20260620 11:37:17.538522 23523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39: Bootstrap starting.
I20260620 11:37:17.539191 23523 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:17.539479 23523 log.cc:824] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:17.540191 23523 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39: No bootstrap required, opened a new log
I20260620 11:37:17.541996 23523 raft_consensus.cc:348] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4168f9d721465e990dd2a2c78d8a39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33801 } }
I20260620 11:37:17.542160 23526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:17.538635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37773 (local address 127.0.0.1:33801)
0620 11:37:17.538844 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:17.538854 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:17.540012 (+ 1158us) server_negotiation.cc:407] Connection header received
0620 11:37:17.540627 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:17.540635 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:17.540664 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:17.540822 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:17.541580 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:17.541584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:17.541584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:17.541589 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:17.541631 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:17.541648 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:17.541757 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:17.541795 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 11:37:17.541836 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
I20260620 11:37:17.542162 23523 raft_consensus.cc:374] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:17.542290 23523 raft_consensus.cc:729] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be4168f9d721465e990dd2a2c78d8a39, State: Initialized, Role: FOLLOWER
I20260620 11:37:17.542418 23523 consensus_queue.cc:260] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4168f9d721465e990dd2a2c78d8a39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33801 } }
I20260620 11:37:17.542537 23523 raft_consensus.cc:388] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:17.542583 23523 raft_consensus.cc:482] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:17.542634 23523 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:17.543241 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:37:17.543424 23523 raft_consensus.cc:504] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4168f9d721465e990dd2a2c78d8a39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33801 } }
I20260620 11:37:17.543524 23523 leader_election.cc:302] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be4168f9d721465e990dd2a2c78d8a39; no voters:
I20260620 11:37:17.543682 23523 leader_election.cc:288] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:17.543733 23528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:17.543879 23528 raft_consensus.cc:686] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39 [term 1 LEADER]: Becoming Leader. State: Replica: be4168f9d721465e990dd2a2c78d8a39, State: Running, Role: LEADER
I20260620 11:37:17.543926 23523 sys_catalog.cc:564] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:17.544042 23528 consensus_queue.cc:237] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4168f9d721465e990dd2a2c78d8a39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33801 } }
I20260620 11:37:17.544492 23530 sys_catalog.cc:455] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be4168f9d721465e990dd2a2c78d8a39. Latest consensus state: current_term: 1 leader_uuid: "be4168f9d721465e990dd2a2c78d8a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4168f9d721465e990dd2a2c78d8a39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33801 } } }
I20260620 11:37:17.544572 23530 sys_catalog.cc:458] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:17.544721 23529 sys_catalog.cc:455] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be4168f9d721465e990dd2a2c78d8a39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be4168f9d721465e990dd2a2c78d8a39" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33801 } } }
I20260620 11:37:17.544788 23529 sys_catalog.cc:458] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:17.544793 23540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:17.546121 23540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:17.546614 23526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:17.545359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42311 (local address 127.0.0.1:33801)
0620 11:37:17.545399 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:17.545401 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:17.545517 (+ 116us) server_negotiation.cc:407] Connection header received
0620 11:37:17.545590 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:17.545591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:17.545596 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:17.545664 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:17.546279 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:17.546281 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:17.546281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:17.546286 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:17.546300 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:17.546304 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:17.546337 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:17.546540 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 11:37:17.546553 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:37:17.547596 23540 catalog_manager.cc:1347] Generated new cluster ID: 0110b683068f44bf995c565a3acb35ca
I20260620 11:37:17.547646 23540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:37:17.563337 23540 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:17.563522 23540 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:17.570152 23540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be4168f9d721465e990dd2a2c78d8a39: Generated new TSK 0
I20260620 11:37:17.570326 23540 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:37:17.571832 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 23455
2026-06-20T11:37:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:17 dist-test-slave-ctbw krb5kdc[23554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:17 dist-test-slave-ctbw krb5kdc[23554](info): set up 2 sockets
Jun 20 11:37:17 dist-test-slave-ctbw krb5kdc[23554](info): commencing operation
krb5kdc: starting...
W20260620 11:37:18.615202 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:18 dist-test-slave-ctbw krb5kdc[23554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:37:18.729972 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44329
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37643
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:18.809882 23567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:18.810084 23567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:18.810106 23567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:18.811317 23567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:18.811358 23567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:18.811373 23567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:18.811385 23567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:18.811429 23567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:18.812887 23567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37643
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44329
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:18.813241 23567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:18.813517 23567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:18.818288 23573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:18.818377 23575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:18.818388 23572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:18.818534 23567 server_base.cc:1034] running on GCE node
I20260620 11:37:18.818780 23567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:18.819147 23567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:18.820297 23567 hybrid_clock.cc:648] HybridClock initialized: now 1781955438820273 us; error 36 us; skew 500 ppm
Jun 20 11:37:18 dist-test-slave-ctbw krb5kdc[23554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955438, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:18.822736 23567 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:37:18.823186 23567 webserver.cc:460] Webserver started at http://127.0.0.1:39899/ using document root <none> and password file <none>
I20260620 11:37:18.823422 23567 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:18.823467 23567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:18.823628 23567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:18.824656 23567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "77e2a0ebb2374eedb24c449813cec766"
format_stamp: "Formatted at 2026-06-20 11:37:18 on dist-test-slave-ctbw"
server_key: "6fc4883c836bdee78b3f728dc56cf720"
server_key_iv: "64a731d67dfdfc8ab8e29e3d6f9b5268"
server_key_version: "encryptionkey@0"
I20260620 11:37:18.824997 23567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "77e2a0ebb2374eedb24c449813cec766"
format_stamp: "Formatted at 2026-06-20 11:37:18 on dist-test-slave-ctbw"
server_key: "6fc4883c836bdee78b3f728dc56cf720"
server_key_iv: "64a731d67dfdfc8ab8e29e3d6f9b5268"
server_key_version: "encryptionkey@0"
I20260620 11:37:18.826269 23567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:37:18.827060 23582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:18.827235 23567 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:37:18.827337 23567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "77e2a0ebb2374eedb24c449813cec766"
format_stamp: "Formatted at 2026-06-20 11:37:18 on dist-test-slave-ctbw"
server_key: "6fc4883c836bdee78b3f728dc56cf720"
server_key_iv: "64a731d67dfdfc8ab8e29e3d6f9b5268"
server_key_version: "encryptionkey@0"
I20260620 11:37:18.827430 23567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:18.843078 23567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:18.845559 23567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:18.845731 23567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:18.849871 23567 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44329
I20260620 11:37:18.849911 23634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44329 every 8 connection(s)
I20260620 11:37:18.850270 23567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:18.851289 23635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:18.853391 23635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766: Bootstrap starting.
I20260620 11:37:18.853897 23635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:18.854142 23635 log.cc:824] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:18.854250 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 23567
I20260620 11:37:18.854444 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:18.854662 3022 external_mini_cluster.cc:1428] Setting key 45eea216a941f4cda11558a7ef46dd0a
I20260620 11:37:18.855006 23635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766: No bootstrap required, opened a new log
I20260620 11:37:18.857174 23635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e2a0ebb2374eedb24c449813cec766" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44329 } }
I20260620 11:37:18.857383 23635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:18.857419 23635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77e2a0ebb2374eedb24c449813cec766, State: Initialized, Role: FOLLOWER
I20260620 11:37:18.857545 23635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e2a0ebb2374eedb24c449813cec766" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44329 } }
I20260620 11:37:18.857635 23635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:18.857681 23635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:18.857734 23635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:18.858372 23635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e2a0ebb2374eedb24c449813cec766" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44329 } }
I20260620 11:37:18.858474 23635 leader_election.cc:302] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77e2a0ebb2374eedb24c449813cec766; no voters:
I20260620 11:37:18.858610 23635 leader_election.cc:288] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:18.858662 23640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:18.858820 23640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766 [term 1 LEADER]: Becoming Leader. State: Replica: 77e2a0ebb2374eedb24c449813cec766, State: Running, Role: LEADER
I20260620 11:37:18.858824 23635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:18.859030 23640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e2a0ebb2374eedb24c449813cec766" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44329 } }
I20260620 11:37:18.860535 23642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77e2a0ebb2374eedb24c449813cec766" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e2a0ebb2374eedb24c449813cec766" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44329 } } }
I20260620 11:37:18.860563 23641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77e2a0ebb2374eedb24c449813cec766. Latest consensus state: current_term: 1 leader_uuid: "77e2a0ebb2374eedb24c449813cec766" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77e2a0ebb2374eedb24c449813cec766" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44329 } } }
I20260620 11:37:18.860672 23642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:18.860718 23641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766 [sys.catalog]: This master's current role is: LEADER
W20260620 11:37:18.860723 23655 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:37:18.860882 23655 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:37:18.861102 23657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:18.861270 23657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:18.862715 23657 catalog_manager.cc:1347] Generated new cluster ID: 9e71b08676b0454d998a990448464f49
I20260620 11:37:18.862776 23657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:37:18 dist-test-slave-ctbw krb5kdc[23554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955438, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 11:37:18.867568 23639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:18.855215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51817 (local address 127.0.0.1:44329)
0620 11:37:18.855384 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:18.855393 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:18.855430 (+ 37us) server_negotiation.cc:407] Connection header received
0620 11:37:18.856419 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:18.856427 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:18.856486 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:18.856630 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:18.857354 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:18.859050 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:18.860223 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:18.860847 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:18.860952 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:18.863948 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:18.863952 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:18.863952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:18.863962 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:18.865864 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:18.866389 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:18.866391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:18.866392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:18.866491 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:18.866714 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:18.866715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:18.866715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:18.867008 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:18.867141 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:18.867273 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 11:37:18.867391 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
I20260620 11:37:18.868691 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:37:18.877013 23639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:18.870123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57631 (local address 127.0.0.1:44329)
0620 11:37:18.870157 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:18.870158 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:18.870166 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:37:18.870188 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:18.870189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:18.870199 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:18.870263 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:18.870820 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:18.871574 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:18.872519 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:18.872850 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:18.872937 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:18.874179 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:18.874180 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:18.874180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:18.874187 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:18.875838 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:18.876217 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:18.876218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:18.876218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:18.876272 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:18.876460 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:18.876461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:18.876461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:18.876576 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:18.876666 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:18.876763 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 11:37:18.876842 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:37:18.895412 23657 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:18.896173 23657 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:18.902833 23657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77e2a0ebb2374eedb24c449813cec766: Generated new TSK 0
I20260620 11:37:18.903030 23657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:37:18.910069 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 23567
2026-06-20T11:37:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1338 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:18 dist-test-slave-ctbw krb5kdc[23666](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:18 dist-test-slave-ctbw krb5kdc[23666](info): set up 2 sockets
Jun 20 11:37:18 dist-test-slave-ctbw krb5kdc[23666](info): commencing operation
krb5kdc: starting...
W20260620 11:37:19.955098 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:20 dist-test-slave-ctbw krb5kdc[23666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955440, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:37:20.075378 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40315
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:20.155465 23679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:20.155709 23679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:20.155746 23679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:20.156986 23679 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:20.157064 23679 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:20.157089 23679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:20.157116 23679 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:20.157146 23679 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:20.158595 23679 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23679
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:20.158895 23679 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:20.159128 23679 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:20.164012 23685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:20.164021 23687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:20.164059 23679 server_base.cc:1034] running on GCE node
W20260620 11:37:20.164021 23684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:20.164480 23679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:20.164793 23679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:20.165982 23679 hybrid_clock.cc:648] HybridClock initialized: now 1781955440165961 us; error 32 us; skew 500 ppm
Jun 20 11:37:20 dist-test-slave-ctbw krb5kdc[23666](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955440, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:20.168610 23679 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:37:20.169073 23679 webserver.cc:460] Webserver started at http://127.0.0.1:39899/ using document root <none> and password file <none>
I20260620 11:37:20.169377 23679 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:20.169461 23679 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:20.169634 23679 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:20.170734 23679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "0959f75c68af4f19b8c1dc50a46b9b63"
format_stamp: "Formatted at 2026-06-20 11:37:20 on dist-test-slave-ctbw"
server_key: "40ee521e6bbc79c92f8334d4247a6357"
server_key_iv: "82b0e56e5838bb1c82cfc213ee1eb750"
server_key_version: "encryptionkey@0"
I20260620 11:37:20.171151 23679 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "0959f75c68af4f19b8c1dc50a46b9b63"
format_stamp: "Formatted at 2026-06-20 11:37:20 on dist-test-slave-ctbw"
server_key: "40ee521e6bbc79c92f8334d4247a6357"
server_key_iv: "82b0e56e5838bb1c82cfc213ee1eb750"
server_key_version: "encryptionkey@0"
I20260620 11:37:20.172546 23679 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:37:20.173373 23694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:20.173548 23679 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:37:20.173658 23679 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "0959f75c68af4f19b8c1dc50a46b9b63"
format_stamp: "Formatted at 2026-06-20 11:37:20 on dist-test-slave-ctbw"
server_key: "40ee521e6bbc79c92f8334d4247a6357"
server_key_iv: "82b0e56e5838bb1c82cfc213ee1eb750"
server_key_version: "encryptionkey@0"
I20260620 11:37:20.173759 23679 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:20.193274 23679 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:20.195808 23679 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:20.195997 23679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:20.200341 23679 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40315
I20260620 11:37:20.200374 23746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40315 every 8 connection(s)
I20260620 11:37:20.200722 23679 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:20.201994 23747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:20.204345 23747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63: Bootstrap starting.
I20260620 11:37:20.204847 23747 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:20.205071 23747 log.cc:824] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:20.205685 23747 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63: No bootstrap required, opened a new log
I20260620 11:37:20.207274 23747 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0959f75c68af4f19b8c1dc50a46b9b63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40315 } }
I20260620 11:37:20.207406 23747 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:20.207429 23747 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0959f75c68af4f19b8c1dc50a46b9b63, State: Initialized, Role: FOLLOWER
I20260620 11:37:20.207561 23747 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0959f75c68af4f19b8c1dc50a46b9b63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40315 } }
I20260620 11:37:20.207638 23747 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:20.207661 23747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:20.207690 23747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:20.208144 23747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0959f75c68af4f19b8c1dc50a46b9b63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40315 } }
I20260620 11:37:20.208225 23747 leader_election.cc:302] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0959f75c68af4f19b8c1dc50a46b9b63; no voters:
I20260620 11:37:20.208379 23747 leader_election.cc:288] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:20.208406 23750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:20.208518 23750 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63 [term 1 LEADER]: Becoming Leader. State: Replica: 0959f75c68af4f19b8c1dc50a46b9b63, State: Running, Role: LEADER
I20260620 11:37:20.208595 23750 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0959f75c68af4f19b8c1dc50a46b9b63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40315 } }
I20260620 11:37:20.208626 23747 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:20.208884 23750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0959f75c68af4f19b8c1dc50a46b9b63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0959f75c68af4f19b8c1dc50a46b9b63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40315 } } }
I20260620 11:37:20.208950 23750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:20.209079 23750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0959f75c68af4f19b8c1dc50a46b9b63. Latest consensus state: current_term: 1 leader_uuid: "0959f75c68af4f19b8c1dc50a46b9b63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0959f75c68af4f19b8c1dc50a46b9b63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40315 } } }
I20260620 11:37:20.209209 23750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:20.209337 23754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:20.209940 23754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:20.210112 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 23679
I20260620 11:37:20.210240 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:20.210422 3022 external_mini_cluster.cc:1428] Setting key 6ac47834419653e305a91efe0e50497d
I20260620 11:37:20.211534 23754 catalog_manager.cc:1347] Generated new cluster ID: 2df36da8ed5a4da39aa5f1c42e06cbf2
I20260620 11:37:20.211593 23754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:37:20.215428 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:20.210831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52565 (local address 127.0.0.1:40315)
0620 11:37:20.211019 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:20.211030 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:20.211115 (+ 85us) server_negotiation.cc:407] Connection header received
0620 11:37:20.211256 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:20.211263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:20.211320 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:20.211427 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:20.211749 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:20.212645 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:20.213473 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:20.213906 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:20.214038 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:20.214734 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:20.214738 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:20.214738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:20.214961 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:20.214996 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:20.215007 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:20.215083 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:20.215155 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 11:37:20.215216 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":82,"threads_started":1}
I20260620 11:37:20.216218 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:37:20.220552 23769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:20.217470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59717 (local address 127.0.0.1:40315)
0620 11:37:20.217505 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:20.217506 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:20.217751 (+ 245us) server_negotiation.cc:407] Connection header received
0620 11:37:20.217834 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:20.217835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:20.217845 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:20.217900 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:20.218006 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:20.218467 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:20.219281 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:20.219499 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:20.219583 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:20.220255 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:20.220256 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:20.220256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:20.220273 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:20.220286 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:20.220297 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:20.220365 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:20.220434 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 11:37:20.220455 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":18}
I20260620 11:37:20.222187 23754 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:20.222680 23754 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:20.226460 23754 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0959f75c68af4f19b8c1dc50a46b9b63: Generated new TSK 0
I20260620 11:37:20.226609 23754 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:37:20.232424 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 23679
2026-06-20T11:37:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1321 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:20 dist-test-slave-ctbw krb5kdc[23778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:20 dist-test-slave-ctbw krb5kdc[23778](info): set up 2 sockets
Jun 20 11:37:20 dist-test-slave-ctbw krb5kdc[23778](info): commencing operation
krb5kdc: starting...
W20260620 11:37:21.271127 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:21 dist-test-slave-ctbw krb5kdc[23778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:37:21.383358 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41127
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:21.462450 23791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:21.462675 23791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:21.462710 23791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:21.463822 23791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:21.463896 23791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:21.463923 23791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:21.463956 23791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:21.463985 23791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:21.465410 23791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:21.465675 23791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:21.465910 23791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:21.470703 23796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:21.470734 23799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:21.470749 23797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:21.470850 23791 server_base.cc:1034] running on GCE node
I20260620 11:37:21.471174 23791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:21.471438 23791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:21.472584 23791 hybrid_clock.cc:648] HybridClock initialized: now 1781955441472560 us; error 41 us; skew 500 ppm
Jun 20 11:37:21 dist-test-slave-ctbw krb5kdc[23778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955441, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:21.474987 23791 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:37:21.475353 23791 webserver.cc:460] Webserver started at http://127.0.0.1:42439/ using document root <none> and password file <none>
I20260620 11:37:21.475543 23791 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:21.475584 23791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:21.475666 23791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:21.476594 23791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "a08b489131b34ad59ab088313afcdb98"
format_stamp: "Formatted at 2026-06-20 11:37:21 on dist-test-slave-ctbw"
server_key: "56c872cf4b305c6fe7a3b5c26bd0d2ab"
server_key_iv: "b90b79145423e94c327a85e115e5eccb"
server_key_version: "encryptionkey@0"
I20260620 11:37:21.476876 23791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "a08b489131b34ad59ab088313afcdb98"
format_stamp: "Formatted at 2026-06-20 11:37:21 on dist-test-slave-ctbw"
server_key: "56c872cf4b305c6fe7a3b5c26bd0d2ab"
server_key_iv: "b90b79145423e94c327a85e115e5eccb"
server_key_version: "encryptionkey@0"
I20260620 11:37:21.478066 23791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 11:37:21.478741 23806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:21.478945 23791 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:37:21.479048 23791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "a08b489131b34ad59ab088313afcdb98"
format_stamp: "Formatted at 2026-06-20 11:37:21 on dist-test-slave-ctbw"
server_key: "56c872cf4b305c6fe7a3b5c26bd0d2ab"
server_key_iv: "b90b79145423e94c327a85e115e5eccb"
server_key_version: "encryptionkey@0"
I20260620 11:37:21.479131 23791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:21.482326 23791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:21.484170 23791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:21.484336 23791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:21.488309 23791 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41127
I20260620 11:37:21.488341 23858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41127 every 8 connection(s)
I20260620 11:37:21.488653 23791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:21.489607 23859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:21.491832 23859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98: Bootstrap starting.
I20260620 11:37:21.492357 23859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:21.492632 23859 log.cc:824] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:21.493268 23859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98: No bootstrap required, opened a new log
I20260620 11:37:21.494793 23859 raft_consensus.cc:348] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08b489131b34ad59ab088313afcdb98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41127 } }
I20260620 11:37:21.494982 23859 raft_consensus.cc:374] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:21.495009 23859 raft_consensus.cc:729] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a08b489131b34ad59ab088313afcdb98, State: Initialized, Role: FOLLOWER
I20260620 11:37:21.495165 23859 consensus_queue.cc:260] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08b489131b34ad59ab088313afcdb98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41127 } }
I20260620 11:37:21.495247 23859 raft_consensus.cc:388] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:21.495298 23859 raft_consensus.cc:482] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:21.495347 23859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:21.495815 23859 raft_consensus.cc:504] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08b489131b34ad59ab088313afcdb98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41127 } }
I20260620 11:37:21.495920 23859 leader_election.cc:302] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a08b489131b34ad59ab088313afcdb98; no voters:
I20260620 11:37:21.496098 23859 leader_election.cc:288] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:21.496155 23862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:21.496330 23862 raft_consensus.cc:686] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98 [term 1 LEADER]: Becoming Leader. State: Replica: a08b489131b34ad59ab088313afcdb98, State: Running, Role: LEADER
I20260620 11:37:21.496351 23859 sys_catalog.cc:564] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:21.496443 23862 consensus_queue.cc:237] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08b489131b34ad59ab088313afcdb98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41127 } }
I20260620 11:37:21.496781 23864 sys_catalog.cc:455] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a08b489131b34ad59ab088313afcdb98. Latest consensus state: current_term: 1 leader_uuid: "a08b489131b34ad59ab088313afcdb98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08b489131b34ad59ab088313afcdb98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41127 } } }
I20260620 11:37:21.496906 23863 sys_catalog.cc:455] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a08b489131b34ad59ab088313afcdb98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a08b489131b34ad59ab088313afcdb98" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41127 } } }
I20260620 11:37:21.496918 23864 sys_catalog.cc:458] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:21.497004 23863 sys_catalog.cc:458] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:21.497332 23871 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:21.497786 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 23791
I20260620 11:37:21.497912 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:21.497967 23871 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:21.498119 3022 external_mini_cluster.cc:1428] Setting key 7ce258e5611a7645cd899fe841faf881
I20260620 11:37:21.499548 23871 catalog_manager.cc:1347] Generated new cluster ID: d75e21d3f40340f988f5ed8c50fb3251
I20260620 11:37:21.499601 23871 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:37:21.508548 23881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:21.498709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40795 (local address 127.0.0.1:41127)
0620 11:37:21.498928 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:21.498936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:21.507070 (+ 8134us) server_negotiation.cc:407] Connection header received
0620 11:37:21.507195 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:21.507201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:21.507252 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:21.507409 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:21.508094 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:21.508098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:21.508098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:21.508102 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:21.508149 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:21.508160 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:21.508264 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:21.508339 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 11:37:21.508383 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":85,"threads_started":1}
I20260620 11:37:21.509516 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:37:21.512018 23881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:21.510973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60125 (local address 127.0.0.1:41127)
0620 11:37:21.511008 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:21.511010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:21.511017 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:37:21.511038 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:21.511039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:21.511043 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:21.511115 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:21.511726 (+ 611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:21.511727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:21.511727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:21.511731 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:21.511743 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:21.511747 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:21.511795 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:21.511846 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 11:37:21.511860 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:37:21.520685 23871 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:21.520859 23871 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:21.525527 23871 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a08b489131b34ad59ab088313afcdb98: Generated new TSK 0
I20260620 11:37:21.525691 23871 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:37:21.527580 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 23791
2026-06-20T11:37:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1294 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:21 dist-test-slave-ctbw krb5kdc[23890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:21 dist-test-slave-ctbw krb5kdc[23890](info): set up 2 sockets
Jun 20 11:37:21 dist-test-slave-ctbw krb5kdc[23890](info): commencing operation
krb5kdc: starting...
W20260620 11:37:22.567205 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:22 dist-test-slave-ctbw krb5kdc[23890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:37:22.682224 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43687
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43687
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:22.761040 23903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:22.761279 23903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:22.761317 23903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:22.762473 23903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:22.762544 23903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:22.762570 23903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:22.762600 23903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:22.762631 23903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:22.764139 23903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:22.764415 23903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:22.764654 23903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:22.769492 23911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:22.769665 23903 server_base.cc:1034] running on GCE node
W20260620 11:37:22.769503 23908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:22.769498 23909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:22.769999 23903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:22.770236 23903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:22.771374 23903 hybrid_clock.cc:648] HybridClock initialized: now 1781955442771356 us; error 41 us; skew 500 ppm
Jun 20 11:37:22 dist-test-slave-ctbw krb5kdc[23890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:22.778126 23903 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:37:22.778580 23903 webserver.cc:460] Webserver started at http://127.0.0.1:41513/ using document root <none> and password file <none>
I20260620 11:37:22.778785 23903 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:22.778826 23903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:22.778988 23903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:22.779991 23903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "4399a88295f3414ebc77546565c1cbbd"
format_stamp: "Formatted at 2026-06-20 11:37:22 on dist-test-slave-ctbw"
server_key: "ec5c74dffbc3d0ae4ec50721649ebabc"
server_key_iv: "d976cd9dcf7307992612dcd038dbadd0"
server_key_version: "encryptionkey@0"
I20260620 11:37:22.780284 23903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "4399a88295f3414ebc77546565c1cbbd"
format_stamp: "Formatted at 2026-06-20 11:37:22 on dist-test-slave-ctbw"
server_key: "ec5c74dffbc3d0ae4ec50721649ebabc"
server_key_iv: "d976cd9dcf7307992612dcd038dbadd0"
server_key_version: "encryptionkey@0"
I20260620 11:37:22.781491 23903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:37:22.782171 23918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:22.782341 23903 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:37:22.782441 23903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "4399a88295f3414ebc77546565c1cbbd"
format_stamp: "Formatted at 2026-06-20 11:37:22 on dist-test-slave-ctbw"
server_key: "ec5c74dffbc3d0ae4ec50721649ebabc"
server_key_iv: "d976cd9dcf7307992612dcd038dbadd0"
server_key_version: "encryptionkey@0"
I20260620 11:37:22.782516 23903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:22.793077 23903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:22.795056 23903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:22.795212 23903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:22.799170 23903 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43687
I20260620 11:37:22.799211 23970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43687 every 8 connection(s)
I20260620 11:37:22.799556 23903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:22.800505 23971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:22.802662 23971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd: Bootstrap starting.
I20260620 11:37:22.803279 23971 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:22.803560 23971 log.cc:824] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:22.804247 23971 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd: No bootstrap required, opened a new log
I20260620 11:37:22.805773 23971 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4399a88295f3414ebc77546565c1cbbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260620 11:37:22.805904 23971 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:22.805927 23971 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4399a88295f3414ebc77546565c1cbbd, State: Initialized, Role: FOLLOWER
I20260620 11:37:22.806100 23971 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4399a88295f3414ebc77546565c1cbbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260620 11:37:22.806170 23971 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:22.806213 23971 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:22.806263 23971 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:22.806500 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 23903
I20260620 11:37:22.806638 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:22.806838 3022 external_mini_cluster.cc:1428] Setting key c6765ef5d1e9fa8464ef2d0b4eb49096
I20260620 11:37:22.806931 23971 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4399a88295f3414ebc77546565c1cbbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260620 11:37:22.807056 23971 leader_election.cc:302] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4399a88295f3414ebc77546565c1cbbd; no voters:
I20260620 11:37:22.807232 23971 leader_election.cc:288] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:22.807367 23974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:22.807511 23974 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd [term 1 LEADER]: Becoming Leader. State: Replica: 4399a88295f3414ebc77546565c1cbbd, State: Running, Role: LEADER
I20260620 11:37:22.807509 23971 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:22.807618 23974 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4399a88295f3414ebc77546565c1cbbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } }
I20260620 11:37:22.807934 23974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4399a88295f3414ebc77546565c1cbbd. Latest consensus state: current_term: 1 leader_uuid: "4399a88295f3414ebc77546565c1cbbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4399a88295f3414ebc77546565c1cbbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } } }
I20260620 11:37:22.808027 23974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:22.807950 23978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4399a88295f3414ebc77546565c1cbbd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4399a88295f3414ebc77546565c1cbbd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43687 } } }
I20260620 11:37:22.808145 23978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:22.808548 23980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:22.809082 23980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:22.810570 23980 catalog_manager.cc:1347] Generated new cluster ID: f40842fe1d324e6cb19f7d4e797e009d
I20260620 11:37:22.810626 23980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:37:22 dist-test-slave-ctbw krb5kdc[23890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955442, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 11:37:22.819880 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:22.807361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:42907 (local address 127.0.0.1:43687)
0620 11:37:22.807708 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:22.807718 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:22.808339 (+ 621us) server_negotiation.cc:407] Connection header received
0620 11:37:22.808566 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:22.808573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:22.808629 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:22.808737 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:22.809701 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:22.811092 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:22.812112 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:22.812760 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:22.812877 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:22.815639 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:22.815643 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:22.815644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:22.815655 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:22.818157 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:22.818606 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:22.818609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:22.818609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:22.818719 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:22.818976 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:22.818977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:22.818977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:22.819278 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:22.819425 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:22.819568 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 11:37:22.819660 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":106,"threads_started":1}
I20260620 11:37:22.820880 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:37:22.829510 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:22.822296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:59647 (local address 127.0.0.1:43687)
0620 11:37:22.822335 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:22.822337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:22.822496 (+ 159us) server_negotiation.cc:407] Connection header received
0620 11:37:22.822549 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:22.822550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:22.822559 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:22.822607 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:22.823228 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:22.823976 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:22.824983 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:22.825314 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:22.825371 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:22.826748 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:22.826749 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:22.826750 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:22.826755 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:22.828298 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:22.828708 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:22.828709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:22.828709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:22.828758 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:22.828984 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:22.828984 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:22.828985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:22.829091 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:22.829183 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:22.829371 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 11:37:22.829428 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":10,"server-negotiator.queue_time_us":20}
I20260620 11:37:22.839293 23980 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:22.839793 23980 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:22.842597 23980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4399a88295f3414ebc77546565c1cbbd: Generated new TSK 0
I20260620 11:37:22.842779 23980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:37:22.852485 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 23903
W20260620 11:37:22.856956 23888 connection.cc:535] client connection to 127.0.0.1:43687 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43687): Connection reset by peer (error 104)
2026-06-20T11:37:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:22 dist-test-slave-ctbw krb5kdc[24002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:22 dist-test-slave-ctbw krb5kdc[24002](info): set up 2 sockets
Jun 20 11:37:22 dist-test-slave-ctbw krb5kdc[24002](info): commencing operation
krb5kdc: starting...
W20260620 11:37:23.903898 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:23 dist-test-slave-ctbw krb5kdc[24002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955443, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:37:24.025931 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40501
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:24.107749 24015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:24.107954 24015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:24.107977 24015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:24.109118 24015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:24.109159 24015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:24.109174 24015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:24.109186 24015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:24.109198 24015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:24.110707 24015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:24.110966 24015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:24.111209 24015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:24.115918 24021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:24.115947 24020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:24.116065 24015 server_base.cc:1034] running on GCE node
W20260620 11:37:24.116020 24023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:24.116351 24015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:24.116638 24015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:24.117833 24015 hybrid_clock.cc:648] HybridClock initialized: now 1781955444117815 us; error 28 us; skew 500 ppm
Jun 20 11:37:24 dist-test-slave-ctbw krb5kdc[24002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955444, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:24.120445 24015 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:37:24.120882 24015 webserver.cc:460] Webserver started at http://127.0.0.1:39935/ using document root <none> and password file <none>
I20260620 11:37:24.121136 24015 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:24.121208 24015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:24.121356 24015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:24.122386 24015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "90380fb36b2c47f491244d3f707d404c"
format_stamp: "Formatted at 2026-06-20 11:37:24 on dist-test-slave-ctbw"
server_key: "e918d9cd1572d3bcb0aa623fa2a5966c"
server_key_iv: "184aed6f9b652e1460023ffd2b1c33c7"
server_key_version: "encryptionkey@0"
I20260620 11:37:24.122745 24015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "90380fb36b2c47f491244d3f707d404c"
format_stamp: "Formatted at 2026-06-20 11:37:24 on dist-test-slave-ctbw"
server_key: "e918d9cd1572d3bcb0aa623fa2a5966c"
server_key_iv: "184aed6f9b652e1460023ffd2b1c33c7"
server_key_version: "encryptionkey@0"
I20260620 11:37:24.124138 24015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:37:24.124903 24030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:24.125101 24015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:37:24.125202 24015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "90380fb36b2c47f491244d3f707d404c"
format_stamp: "Formatted at 2026-06-20 11:37:24 on dist-test-slave-ctbw"
server_key: "e918d9cd1572d3bcb0aa623fa2a5966c"
server_key_iv: "184aed6f9b652e1460023ffd2b1c33c7"
server_key_version: "encryptionkey@0"
I20260620 11:37:24.125284 24015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:24.135167 24015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:24.137478 24015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:24.137637 24015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:24.141506 24015 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40501
I20260620 11:37:24.141579 24082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40501 every 8 connection(s)
I20260620 11:37:24.141924 24015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:24.142953 24083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:24.145159 24083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c: Bootstrap starting.
I20260620 11:37:24.145723 24083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:24.145998 24083 log.cc:824] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:24.146672 24083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c: No bootstrap required, opened a new log
I20260620 11:37:24.148236 24083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90380fb36b2c47f491244d3f707d404c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40501 } }
I20260620 11:37:24.148367 24083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:24.148391 24083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90380fb36b2c47f491244d3f707d404c, State: Initialized, Role: FOLLOWER
I20260620 11:37:24.148538 24083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90380fb36b2c47f491244d3f707d404c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40501 } }
I20260620 11:37:24.148622 24083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:24.148664 24083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:24.148715 24083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:24.149225 24083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90380fb36b2c47f491244d3f707d404c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40501 } }
I20260620 11:37:24.149329 24083 leader_election.cc:302] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90380fb36b2c47f491244d3f707d404c; no voters:
I20260620 11:37:24.149502 24083 leader_election.cc:288] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:24.149565 24086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:24.149742 24086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c [term 1 LEADER]: Becoming Leader. State: Replica: 90380fb36b2c47f491244d3f707d404c, State: Running, Role: LEADER
I20260620 11:37:24.149832 24083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:24.149928 24086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90380fb36b2c47f491244d3f707d404c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40501 } }
I20260620 11:37:24.150454 24087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90380fb36b2c47f491244d3f707d404c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90380fb36b2c47f491244d3f707d404c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40501 } } }
I20260620 11:37:24.150535 24087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:24.150776 24095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:24.150910 24088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90380fb36b2c47f491244d3f707d404c. Latest consensus state: current_term: 1 leader_uuid: "90380fb36b2c47f491244d3f707d404c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90380fb36b2c47f491244d3f707d404c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40501 } } }
I20260620 11:37:24.150985 24088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:24.151087 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 24015
I20260620 11:37:24.151201 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:24.151386 24095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:24.151403 3022 external_mini_cluster.cc:1428] Setting key c332f3e73f58f9969a804815888fbc46
I20260620 11:37:24.153128 24095 catalog_manager.cc:1347] Generated new cluster ID: cb8689342fb34eb2b5e60866bf9533af
I20260620 11:37:24.153190 24095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:37:24.157390 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:24.151946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:51015 (local address 127.0.0.1:40501)
0620 11:37:24.152173 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:24.152181 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:24.152210 (+ 29us) server_negotiation.cc:407] Connection header received
0620 11:37:24.152320 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:24.152327 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:24.152384 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:24.152521 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:24.152768 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:24.154153 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:24.155299 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:24.155886 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:24.155988 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:24.156716 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:24.156720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:24.156720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:24.156920 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:24.156954 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:24.156967 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:24.157045 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:24.157173 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 11:37:24.157199 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":71,"threads_started":1}
I20260620 11:37:24.158404 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:37:24.166651 24105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:24.163198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:44295 (local address 127.0.0.1:40501)
0620 11:37:24.163234 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:24.163236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:24.163322 (+ 86us) server_negotiation.cc:407] Connection header received
0620 11:37:24.163349 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:24.163350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:24.163360 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:24.163428 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:24.163523 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:24.164303 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:24.165307 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:24.165647 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:24.165717 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:24.166401 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:24.166402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:24.166403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:24.166407 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:24.166423 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:24.166427 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:24.166490 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:24.166562 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 11:37:24.166575 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:37:24.170992 24095 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:24.171484 24095 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:24.184969 24095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90380fb36b2c47f491244d3f707d404c: Generated new TSK 0
I20260620 11:37:24.185211 24095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:37:24.192652 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 24015
W20260620 11:37:24.197036 24000 connection.cc:535] client connection to 127.0.0.1:40501 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40501): Connection reset by peer (error 104)
2026-06-20T11:37:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:24 dist-test-slave-ctbw krb5kdc[24114](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:24 dist-test-slave-ctbw krb5kdc[24114](info): set up 2 sockets
Jun 20 11:37:24 dist-test-slave-ctbw krb5kdc[24114](info): commencing operation
krb5kdc: starting...
W20260620 11:37:25.223318 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:25 dist-test-slave-ctbw krb5kdc[24114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:37:25.335832 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38421
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37739
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:25.415685 24127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:25.415879 24127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:25.415899 24127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:25.417042 24127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:25.417084 24127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:25.417098 24127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:25.417135 24127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:25.417161 24127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:25.418588 24127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37739
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:25.418825 24127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:25.419085 24127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:25.423650 24133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:25.423714 24132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:25.423640 24135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:25.423849 24127 server_base.cc:1034] running on GCE node
I20260620 11:37:25.424093 24127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:25.424314 24127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:25.425457 24127 hybrid_clock.cc:648] HybridClock initialized: now 1781955445425439 us; error 33 us; skew 500 ppm
Jun 20 11:37:25 dist-test-slave-ctbw krb5kdc[24114](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:25.428004 24127 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:37:25.428421 24127 webserver.cc:460] Webserver started at http://127.0.0.1:33989/ using document root <none> and password file <none>
I20260620 11:37:25.428658 24127 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:25.428702 24127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:25.428864 24127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:25.429956 24127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "838ab20fe9024698ac54250ac7f38db7"
format_stamp: "Formatted at 2026-06-20 11:37:25 on dist-test-slave-ctbw"
server_key: "e0e4fb3bfd37903f4a3699f0522395b3"
server_key_iv: "f8c915d2cec222aa94f1f9501217a76a"
server_key_version: "encryptionkey@0"
I20260620 11:37:25.430295 24127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "838ab20fe9024698ac54250ac7f38db7"
format_stamp: "Formatted at 2026-06-20 11:37:25 on dist-test-slave-ctbw"
server_key: "e0e4fb3bfd37903f4a3699f0522395b3"
server_key_iv: "f8c915d2cec222aa94f1f9501217a76a"
server_key_version: "encryptionkey@0"
I20260620 11:37:25.431630 24127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:37:25.432346 24142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:25.432511 24127 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:25.432621 24127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "838ab20fe9024698ac54250ac7f38db7"
format_stamp: "Formatted at 2026-06-20 11:37:25 on dist-test-slave-ctbw"
server_key: "e0e4fb3bfd37903f4a3699f0522395b3"
server_key_iv: "f8c915d2cec222aa94f1f9501217a76a"
server_key_version: "encryptionkey@0"
I20260620 11:37:25.432693 24127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:25.435977 24127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:25.437840 24127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:25.437996 24127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:25.441923 24127 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38421
I20260620 11:37:25.441952 24194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38421 every 8 connection(s)
I20260620 11:37:25.442252 24127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:25.443193 24195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:25.445161 24195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7: Bootstrap starting.
I20260620 11:37:25.445657 24195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:25.445874 24195 log.cc:824] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:25.446466 24195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7: No bootstrap required, opened a new log
I20260620 11:37:25.448019 24195 raft_consensus.cc:348] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838ab20fe9024698ac54250ac7f38db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38421 } }
I20260620 11:37:25.448144 24195 raft_consensus.cc:374] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:25.448165 24195 raft_consensus.cc:729] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 838ab20fe9024698ac54250ac7f38db7, State: Initialized, Role: FOLLOWER
I20260620 11:37:25.448256 24195 consensus_queue.cc:260] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838ab20fe9024698ac54250ac7f38db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38421 } }
I20260620 11:37:25.448305 24195 raft_consensus.cc:388] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:25.448324 24195 raft_consensus.cc:482] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:25.448350 24195 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:25.448794 24195 raft_consensus.cc:504] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838ab20fe9024698ac54250ac7f38db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38421 } }
I20260620 11:37:25.448870 24195 leader_election.cc:302] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 838ab20fe9024698ac54250ac7f38db7; no voters:
I20260620 11:37:25.448983 24195 leader_election.cc:288] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:25.449060 24198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:25.449230 24198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7 [term 1 LEADER]: Becoming Leader. State: Replica: 838ab20fe9024698ac54250ac7f38db7, State: Running, Role: LEADER
I20260620 11:37:25.449268 24195 sys_catalog.cc:564] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:25.449343 24198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838ab20fe9024698ac54250ac7f38db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38421 } }
I20260620 11:37:25.449918 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 24127
I20260620 11:37:25.449914 24200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 838ab20fe9024698ac54250ac7f38db7. Latest consensus state: current_term: 1 leader_uuid: "838ab20fe9024698ac54250ac7f38db7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838ab20fe9024698ac54250ac7f38db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38421 } } }
I20260620 11:37:25.449990 24200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:25.450057 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:25.450244 3022 external_mini_cluster.cc:1428] Setting key caced111d71dba15601cb3da7809bf99
I20260620 11:37:25.450369 24207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:25.450757 24199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "838ab20fe9024698ac54250ac7f38db7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "838ab20fe9024698ac54250ac7f38db7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38421 } } }
I20260620 11:37:25.450914 24199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:25.451022 24207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:25.452672 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:25.450781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:54633 (local address 127.0.0.1:38421)
0620 11:37:25.451002 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:25.451011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:25.451047 (+ 36us) server_negotiation.cc:407] Connection header received
0620 11:37:25.451158 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:25.451164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:25.451192 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:25.451323 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:25.452242 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:25.452245 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:25.452245 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:25.452250 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:25.452289 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:25.452304 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:25.452367 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:25.452454 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 11:37:25.452501 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":103,"threads_started":1}
I20260620 11:37:25.453747 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:37:25.455286 24207 catalog_manager.cc:1347] Generated new cluster ID: 2ca020d9f3224df09c809a0a92a6a5e9
I20260620 11:37:25.455333 24207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:37:25.456238 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:25.455066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:50591 (local address 127.0.0.1:38421)
0620 11:37:25.455106 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:25.455107 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:25.455231 (+ 124us) server_negotiation.cc:407] Connection header received
0620 11:37:25.455342 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:25.455343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:25.455348 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:25.455418 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:25.455999 (+ 581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:25.456001 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:25.456001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:25.456005 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:25.456027 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:25.456031 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:25.456086 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:25.456158 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 11:37:25.456169 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:37:25.468277 24207 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:25.468431 24207 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:25.471619 24207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 838ab20fe9024698ac54250ac7f38db7: Generated new TSK 0
I20260620 11:37:25.471779 24207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:37:25.476089 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 24127
2026-06-20T11:37:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1283 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:25 dist-test-slave-ctbw krb5kdc[24226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:25 dist-test-slave-ctbw krb5kdc[24226](info): set up 2 sockets
Jun 20 11:37:25 dist-test-slave-ctbw krb5kdc[24226](info): commencing operation
krb5kdc: starting...
W20260620 11:37:26.511170 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:26 dist-test-slave-ctbw krb5kdc[24226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:37:26.623299 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:26.704449 24239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:26.704650 24239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:26.704674 24239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:26.705749 24239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:26.705783 24239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:26.705797 24239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:26.705809 24239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:26.705822 24239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:26.707298 24239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41475
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:26.707487 24239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:26.707648 24239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:26.712119 24244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:26.712150 24247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:26.712357 24239 server_base.cc:1034] running on GCE node
W20260620 11:37:26.712152 24245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:26.712625 24239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:26.712857 24239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:26.713995 24239 hybrid_clock.cc:648] HybridClock initialized: now 1781955446713979 us; error 34 us; skew 500 ppm
Jun 20 11:37:26 dist-test-slave-ctbw krb5kdc[24226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955446, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:26.716353 24239 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:37:26.716813 24239 webserver.cc:460] Webserver started at http://127.0.0.1:40959/ using document root <none> and password file <none>
I20260620 11:37:26.717037 24239 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:26.717108 24239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:26.717258 24239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:26.718277 24239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "dd8b7d3392b74e8eafd52b50d6292b4f"
format_stamp: "Formatted at 2026-06-20 11:37:26 on dist-test-slave-ctbw"
server_key: "2320f37909ce9342557e632fde541490"
server_key_iv: "22979f626d4f48d2576a5a76ad6006ce"
server_key_version: "encryptionkey@0"
I20260620 11:37:26.718638 24239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "dd8b7d3392b74e8eafd52b50d6292b4f"
format_stamp: "Formatted at 2026-06-20 11:37:26 on dist-test-slave-ctbw"
server_key: "2320f37909ce9342557e632fde541490"
server_key_iv: "22979f626d4f48d2576a5a76ad6006ce"
server_key_version: "encryptionkey@0"
I20260620 11:37:26.719993 24239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 11:37:26.720682 24254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:26.720840 24239 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:26.720940 24239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "dd8b7d3392b74e8eafd52b50d6292b4f"
format_stamp: "Formatted at 2026-06-20 11:37:26 on dist-test-slave-ctbw"
server_key: "2320f37909ce9342557e632fde541490"
server_key_iv: "22979f626d4f48d2576a5a76ad6006ce"
server_key_version: "encryptionkey@0"
I20260620 11:37:26.721030 24239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:26.732718 24239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:26.734511 24239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:26.734673 24239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:26.738531 24239 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41475
I20260620 11:37:26.738572 24306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41475 every 8 connection(s)
I20260620 11:37:26.738977 24239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:26.739985 24307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:26.742323 24307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f: Bootstrap starting.
I20260620 11:37:26.742955 24307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:26.743203 24307 log.cc:824] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:26.743870 24307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f: No bootstrap required, opened a new log
I20260620 11:37:26.745543 24307 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8b7d3392b74e8eafd52b50d6292b4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41475 } }
I20260620 11:37:26.745685 24307 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:26.745709 24307 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8b7d3392b74e8eafd52b50d6292b4f, State: Initialized, Role: FOLLOWER
I20260620 11:37:26.745817 24307 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8b7d3392b74e8eafd52b50d6292b4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41475 } }
I20260620 11:37:26.745875 24307 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:26.745898 24307 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:26.745930 24307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:26.746428 24307 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8b7d3392b74e8eafd52b50d6292b4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41475 } }
I20260620 11:37:26.746513 24307 leader_election.cc:302] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd8b7d3392b74e8eafd52b50d6292b4f; no voters:
I20260620 11:37:26.746636 24307 leader_election.cc:288] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:26.746703 24310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:26.746847 24310 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f [term 1 LEADER]: Becoming Leader. State: Replica: dd8b7d3392b74e8eafd52b50d6292b4f, State: Running, Role: LEADER
I20260620 11:37:26.746922 24307 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:26.747040 24310 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8b7d3392b74e8eafd52b50d6292b4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41475 } }
I20260620 11:37:26.747646 24311 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd8b7d3392b74e8eafd52b50d6292b4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8b7d3392b74e8eafd52b50d6292b4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41475 } } }
I20260620 11:37:26.747865 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 24239
I20260620 11:37:26.748011 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:26.748168 24311 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:26.748250 3022 external_mini_cluster.cc:1428] Setting key 090ad95323e4b9687f544905f47e3eba
I20260620 11:37:26.748059 24312 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd8b7d3392b74e8eafd52b50d6292b4f. Latest consensus state: current_term: 1 leader_uuid: "dd8b7d3392b74e8eafd52b50d6292b4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8b7d3392b74e8eafd52b50d6292b4f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41475 } } }
I20260620 11:37:26.748294 24312 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f [sys.catalog]: This master's current role is: LEADER
W20260620 11:37:26.749068 24325 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:37:26.749123 24325 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:37:26.749186 24324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:26.749384 24324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:26.750945 24324 catalog_manager.cc:1347] Generated new cluster ID: bc453c650ae64c9194fc86d37302b566
I20260620 11:37:26.751005 24324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:37:26.756007 24324 catalog_manager.cc:1370] Generated new certificate authority record
Jun 20 11:37:26 dist-test-slave-ctbw krb5kdc[24226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 11:37:26.756630 24324 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:26.760681 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:26.748825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:35527 (local address 127.0.0.1:41475)
0620 11:37:26.749256 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:26.749266 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:26.749299 (+ 33us) server_negotiation.cc:407] Connection header received
0620 11:37:26.749426 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:26.749433 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:26.749498 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:26.749658 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:26.750448 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:26.752036 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:26.753148 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:26.753795 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:26.753908 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:26.756828 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:26.756832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:26.756833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:26.756844 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:26.758918 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:26.759382 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:26.759385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:26.759385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:26.759492 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:26.759740 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:26.759741 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:26.759741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:26.760041 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:26.760203 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:26.760332 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 11:37:26.760444 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":86,"threads_started":1}
I20260620 11:37:26.761718 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:37:26.763454 24324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd8b7d3392b74e8eafd52b50d6292b4f: Generated new TSK 0
I20260620 11:37:26.763648 24324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:37:26.770090 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:26.763222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:46951 (local address 127.0.0.1:41475)
0620 11:37:26.763259 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:26.763261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:26.763427 (+ 166us) server_negotiation.cc:407] Connection header received
0620 11:37:26.763491 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:26.763491 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:26.763501 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:26.763559 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:26.764230 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:26.764803 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:26.765854 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:26.766097 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:26.766157 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:26.767520 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:26.767521 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:26.767521 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:26.767528 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:26.768939 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:26.769353 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:26.769354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:26.769354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:26.769422 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:26.769660 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:26.769660 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:26.769660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:26.769755 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:26.769831 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:26.769983 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 11:37:26.770032 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:37:26.771911 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 24239
W20260620 11:37:26.775878 24224 connection.cc:535] client connection to 127.0.0.1:41475 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41475): Connection reset by peer (error 104)
2026-06-20T11:37:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:26 dist-test-slave-ctbw krb5kdc[24338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:26 dist-test-slave-ctbw krb5kdc[24338](info): set up 2 sockets
Jun 20 11:37:26 dist-test-slave-ctbw krb5kdc[24338](info): commencing operation
krb5kdc: starting...
W20260620 11:37:27.807281 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:27 dist-test-slave-ctbw krb5kdc[24338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955447, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:37:27.921036 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42261
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:28.001051 24351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:28.001281 24351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:28.001328 24351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:28.002485 24351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:28.002549 24351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:28.002573 24351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:28.002594 24351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:28.002624 24351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:28.004103 24351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42261
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:28.004382 24351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:28.004612 24351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:28.009423 24356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:28.009438 24357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:28.009438 24359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:28.009742 24351 server_base.cc:1034] running on GCE node
I20260620 11:37:28.009999 24351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:28.010269 24351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:28.011437 24351 hybrid_clock.cc:648] HybridClock initialized: now 1781955448011405 us; error 58 us; skew 500 ppm
Jun 20 11:37:28 dist-test-slave-ctbw krb5kdc[24338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:28.017105 24351 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:37:28.017541 24351 webserver.cc:460] Webserver started at http://127.0.0.1:42781/ using document root <none> and password file <none>
I20260620 11:37:28.017711 24351 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:28.017750 24351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:28.017832 24351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:28.018800 24351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "17a448fb90094922bd2bb00711964a01"
format_stamp: "Formatted at 2026-06-20 11:37:28 on dist-test-slave-ctbw"
server_key: "bb5f1b31a2734a13a9b03619c27822f8"
server_key_iv: "9c374ff03be0cd2a53265b712a4823ec"
server_key_version: "encryptionkey@0"
I20260620 11:37:28.019164 24351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "17a448fb90094922bd2bb00711964a01"
format_stamp: "Formatted at 2026-06-20 11:37:28 on dist-test-slave-ctbw"
server_key: "bb5f1b31a2734a13a9b03619c27822f8"
server_key_iv: "9c374ff03be0cd2a53265b712a4823ec"
server_key_version: "encryptionkey@0"
I20260620 11:37:28.020462 24351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:37:28.021108 24366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:28.021277 24351 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:37:28.021360 24351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "17a448fb90094922bd2bb00711964a01"
format_stamp: "Formatted at 2026-06-20 11:37:28 on dist-test-slave-ctbw"
server_key: "bb5f1b31a2734a13a9b03619c27822f8"
server_key_iv: "9c374ff03be0cd2a53265b712a4823ec"
server_key_version: "encryptionkey@0"
I20260620 11:37:28.021428 24351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:28.047550 24351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:28.049866 24351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:28.049985 24351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:28.053735 24351 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42261
I20260620 11:37:28.053805 24418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42261 every 8 connection(s)
I20260620 11:37:28.054114 24351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:28.055058 24419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:28.055616 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 24351
I20260620 11:37:28.055714 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:28.055979 3022 external_mini_cluster.cc:1428] Setting key 9175311b88596039839a1c33e85208d2
I20260620 11:37:28.058295 24419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01: Bootstrap starting.
I20260620 11:37:28.059013 24419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:28.059293 24419 log.cc:824] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:28.060055 24419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01: No bootstrap required, opened a new log
I20260620 11:37:28.062112 24422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:28.056569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:50305 (local address 127.0.0.1:42261)
0620 11:37:28.056778 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:28.056788 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:28.056817 (+ 29us) server_negotiation.cc:407] Connection header received
0620 11:37:28.057393 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:28.057401 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:28.057506 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:28.057656 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:28.057895 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:28.059259 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:28.060030 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:28.060614 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:28.060726 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:28.061375 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:28.061379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:28.061379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:28.061583 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:28.061625 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:28.061644 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:28.061736 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:28.061814 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 11:37:28.061853 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":73,"threads_started":1}
I20260620 11:37:28.062357 24419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a448fb90094922bd2bb00711964a01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42261 } }
I20260620 11:37:28.062506 24419 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:28.062552 24419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17a448fb90094922bd2bb00711964a01, State: Initialized, Role: FOLLOWER
I20260620 11:37:28.062686 24419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a448fb90094922bd2bb00711964a01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42261 } }
I20260620 11:37:28.062770 24419 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:28.062812 24419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:28.062880 24419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:28.063191 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:37:28.063561 24419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a448fb90094922bd2bb00711964a01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42261 } }
I20260620 11:37:28.063679 24419 leader_election.cc:302] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17a448fb90094922bd2bb00711964a01; no voters:
I20260620 11:37:28.063877 24419 leader_election.cc:288] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:28.063902 24424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:28.064077 24424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01 [term 1 LEADER]: Becoming Leader. State: Replica: 17a448fb90094922bd2bb00711964a01, State: Running, Role: LEADER
I20260620 11:37:28.064172 24419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:28.064239 24424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a448fb90094922bd2bb00711964a01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42261 } }
I20260620 11:37:28.064615 24424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17a448fb90094922bd2bb00711964a01. Latest consensus state: current_term: 1 leader_uuid: "17a448fb90094922bd2bb00711964a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a448fb90094922bd2bb00711964a01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42261 } } }
I20260620 11:37:28.064625 24425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17a448fb90094922bd2bb00711964a01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17a448fb90094922bd2bb00711964a01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42261 } } }
I20260620 11:37:28.064851 24425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:28.065025 24424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:28.065101 24438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:28.065622 24438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:28.067153 24438 catalog_manager.cc:1347] Generated new cluster ID: 157b631a5cea4150943814802cb5f86a
I20260620 11:37:28.067226 24438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:37:28.069384 24422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:28.064574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:43861 (local address 127.0.0.1:42261)
0620 11:37:28.064611 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:28.064613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:28.065681 (+ 1068us) server_negotiation.cc:407] Connection header received
0620 11:37:28.065916 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:28.065917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:28.065927 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:28.065955 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:28.066148 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:28.066849 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:28.068052 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:28.068348 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:28.068423 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:28.069110 (+ 687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:28.069111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:28.069111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:28.069116 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:28.069129 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:28.069133 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:28.069188 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:28.069307 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 11:37:28.069317 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:37:28.080730 24438 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:28.081437 24438 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:28.091022 24438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17a448fb90094922bd2bb00711964a01: Generated new TSK 0
I20260620 11:37:28.091200 24438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:37:28.102465 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 24351
W20260620 11:37:28.107023 24336 connection.cc:535] client connection to 127.0.0.1:42261 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42261): Connection reset by peer (error 104)
2026-06-20T11:37:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:28 dist-test-slave-ctbw krb5kdc[24450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:28 dist-test-slave-ctbw krb5kdc[24450](info): set up 2 sockets
Jun 20 11:37:28 dist-test-slave-ctbw krb5kdc[24450](info): commencing operation
krb5kdc: starting...
W20260620 11:37:29.143215 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:29 dist-test-slave-ctbw krb5kdc[24450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:37:29.253580 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45929
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:29.332137 24463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:29.332331 24463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:29.332353 24463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:29.333412 24463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:29.333446 24463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:29.333460 24463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:29.333473 24463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:29.333485 24463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:29.334901 24463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45929
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:29.335079 24463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:29.335237 24463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:29.339887 24469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:29.339920 24471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:29.339882 24468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:29.340234 24463 server_base.cc:1034] running on GCE node
I20260620 11:37:29.340453 24463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:29.340682 24463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:29.341833 24463 hybrid_clock.cc:648] HybridClock initialized: now 1781955449341817 us; error 35 us; skew 500 ppm
Jun 20 11:37:29 dist-test-slave-ctbw krb5kdc[24450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:29.344352 24463 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:37:29.344780 24463 webserver.cc:460] Webserver started at http://127.0.0.1:40947/ using document root <none> and password file <none>
I20260620 11:37:29.344971 24463 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:29.345011 24463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:29.345096 24463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:29.346071 24463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "1f9dcfce43194850bbba63256d689bf2"
format_stamp: "Formatted at 2026-06-20 11:37:29 on dist-test-slave-ctbw"
server_key: "bd3ba2c0676c59c26bdd411609ebfdd8"
server_key_iv: "2a0a1dc84bf6877ae6c4e2f4cce14327"
server_key_version: "encryptionkey@0"
I20260620 11:37:29.346364 24463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "1f9dcfce43194850bbba63256d689bf2"
format_stamp: "Formatted at 2026-06-20 11:37:29 on dist-test-slave-ctbw"
server_key: "bd3ba2c0676c59c26bdd411609ebfdd8"
server_key_iv: "2a0a1dc84bf6877ae6c4e2f4cce14327"
server_key_version: "encryptionkey@0"
I20260620 11:37:29.347645 24463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:37:29.348316 24478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:29.348485 24463 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:37:29.348591 24463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "1f9dcfce43194850bbba63256d689bf2"
format_stamp: "Formatted at 2026-06-20 11:37:29 on dist-test-slave-ctbw"
server_key: "bd3ba2c0676c59c26bdd411609ebfdd8"
server_key_iv: "2a0a1dc84bf6877ae6c4e2f4cce14327"
server_key_version: "encryptionkey@0"
I20260620 11:37:29.348665 24463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:29.351997 24463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:29.354120 24463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:29.354286 24463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:29.358202 24463 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45929
I20260620 11:37:29.358229 24530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45929 every 8 connection(s)
I20260620 11:37:29.358554 24463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:29.359530 24531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:29.361634 24531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2: Bootstrap starting.
I20260620 11:37:29.362141 24531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:29.362381 24531 log.cc:824] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:29.363093 24531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2: No bootstrap required, opened a new log
I20260620 11:37:29.364570 24531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f9dcfce43194850bbba63256d689bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45929 } }
I20260620 11:37:29.364696 24531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:29.364717 24531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f9dcfce43194850bbba63256d689bf2, State: Initialized, Role: FOLLOWER
I20260620 11:37:29.364818 24531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f9dcfce43194850bbba63256d689bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45929 } }
I20260620 11:37:29.364867 24531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:29.364886 24531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:29.364916 24531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:29.365360 24531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f9dcfce43194850bbba63256d689bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45929 } }
I20260620 11:37:29.365437 24531 leader_election.cc:302] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f9dcfce43194850bbba63256d689bf2; no voters:
I20260620 11:37:29.365548 24531 leader_election.cc:288] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:29.365631 24534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:29.365810 24534 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2 [term 1 LEADER]: Becoming Leader. State: Replica: 1f9dcfce43194850bbba63256d689bf2, State: Running, Role: LEADER
I20260620 11:37:29.365839 24531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:29.365896 24534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f9dcfce43194850bbba63256d689bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45929 } }
I20260620 11:37:29.366237 24536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f9dcfce43194850bbba63256d689bf2. Latest consensus state: current_term: 1 leader_uuid: "1f9dcfce43194850bbba63256d689bf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f9dcfce43194850bbba63256d689bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45929 } } }
I20260620 11:37:29.366232 24535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f9dcfce43194850bbba63256d689bf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f9dcfce43194850bbba63256d689bf2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45929 } } }
I20260620 11:37:29.366338 24536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:29.366343 24535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:29.366532 24538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:29.367216 24538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:29.367818 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 24463
I20260620 11:37:29.367944 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:29.368117 3022 external_mini_cluster.cc:1428] Setting key 971188ea4d4673e841f76b3c23c1d7f2
I20260620 11:37:29.368808 24538 catalog_manager.cc:1347] Generated new cluster ID: 09105ad8acbc49a4bcfcaf3da28e9712
I20260620 11:37:29.368860 24538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:37:29.371932 24554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:29.368765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:39177 (local address 127.0.0.1:45929)
0620 11:37:29.368938 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:29.368948 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:29.368978 (+ 30us) server_negotiation.cc:407] Connection header received
0620 11:37:29.369087 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:29.369093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:29.369117 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:29.369260 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:29.371387 (+ 2127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:29.371391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:29.371391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:29.371395 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:29.371455 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:29.371466 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:29.371541 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:29.371624 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 11:37:29.371669 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260620 11:37:29.372798 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:37:29.375301 24554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:29.374164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.68:60373 (local address 127.0.0.1:45929)
0620 11:37:29.374200 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:29.374202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:29.374337 (+ 135us) server_negotiation.cc:407] Connection header received
0620 11:37:29.374399 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:29.374399 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:29.374405 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:29.374454 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:29.375070 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:29.375072 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:29.375072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:29.375076 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:29.375090 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:29.375094 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:29.375151 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:29.375206 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 11:37:29.375222 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:37:29.380728 24538 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:29.380884 24538 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:29.384720 24538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f9dcfce43194850bbba63256d689bf2: Generated new TSK 0
I20260620 11:37:29.384904 24538 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:37:29.389305 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 24463
2026-06-20T11:37:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1286 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:29 dist-test-slave-ctbw krb5kdc[24562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:29 dist-test-slave-ctbw krb5kdc[24562](info): set up 2 sockets
Jun 20 11:37:29 dist-test-slave-ctbw krb5kdc[24562](info): commencing operation
krb5kdc: starting...
W20260620 11:37:30.419178 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:30 dist-test-slave-ctbw krb5kdc[24562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:37:30.529485 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:36871
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:43571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:36871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:30.608707 24575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:30.608888 24575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:30.608911 24575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:30.610065 24575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:30.610121 24575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:30.610134 24575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:30.610146 24575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:30.610158 24575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:30.611714 24575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:43571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:36871
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:36871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:30.611968 24575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:30.612179 24575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:30.616954 24583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:30.617089 24575 server_base.cc:1034] running on GCE node
W20260620 11:37:30.616976 24580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:30.617204 24581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:30.617424 24575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:30.617748 24575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:30.618958 24575 hybrid_clock.cc:648] HybridClock initialized: now 1781955450618939 us; error 45 us; skew 500 ppm
Jun 20 11:37:30 dist-test-slave-ctbw krb5kdc[24562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955450, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:30.621555 24575 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:37:30.622013 24575 webserver.cc:460] Webserver started at http://127.2.243.190:38353/ using document root <none> and password file <none>
I20260620 11:37:30.622253 24575 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:30.622327 24575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:30.622480 24575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:30.623590 24575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "298faf82f0c644ed87943af77f6dfc95"
format_stamp: "Formatted at 2026-06-20 11:37:30 on dist-test-slave-ctbw"
server_key: "bc05e6801ccb04c3438221c0d65330f6"
server_key_iv: "6cc747220c5168b79aecc04d165aaff2"
server_key_version: "encryptionkey@0"
I20260620 11:37:30.623936 24575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "298faf82f0c644ed87943af77f6dfc95"
format_stamp: "Formatted at 2026-06-20 11:37:30 on dist-test-slave-ctbw"
server_key: "bc05e6801ccb04c3438221c0d65330f6"
server_key_iv: "6cc747220c5168b79aecc04d165aaff2"
server_key_version: "encryptionkey@0"
I20260620 11:37:30.625195 24575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:37:30.625874 24590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:30.626042 24575 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:37:30.626138 24575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "298faf82f0c644ed87943af77f6dfc95"
format_stamp: "Formatted at 2026-06-20 11:37:30 on dist-test-slave-ctbw"
server_key: "bc05e6801ccb04c3438221c0d65330f6"
server_key_iv: "6cc747220c5168b79aecc04d165aaff2"
server_key_version: "encryptionkey@0"
I20260620 11:37:30.626219 24575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:30.643637 24575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:30.646142 24575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:30.646297 24575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:30.650185 24575 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:36871
I20260620 11:37:30.650242 24642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:36871 every 8 connection(s)
I20260620 11:37:30.650542 24575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:30.651393 24643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:30.653650 24643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95: Bootstrap starting.
I20260620 11:37:30.653673 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 24575
I20260620 11:37:30.653787 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:30.653968 3022 external_mini_cluster.cc:1428] Setting key 962fccaa36e12ee969a80beafc791adc
I20260620 11:37:30.654371 24643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:30.654711 24643 log.cc:824] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:30.655455 24643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95: No bootstrap required, opened a new log
I20260620 11:37:30.657722 24643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298faf82f0c644ed87943af77f6dfc95" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36871 } }
I20260620 11:37:30.657887 24643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:30.657922 24643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 298faf82f0c644ed87943af77f6dfc95, State: Initialized, Role: FOLLOWER
I20260620 11:37:30.658036 24643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298faf82f0c644ed87943af77f6dfc95" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36871 } }
I20260620 11:37:30.658103 24643 raft_consensus.cc:388] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:30.658129 24643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:30.658172 24643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:30.658725 24643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298faf82f0c644ed87943af77f6dfc95" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36871 } }
I20260620 11:37:30.658810 24643 leader_election.cc:302] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 298faf82f0c644ed87943af77f6dfc95; no voters:
I20260620 11:37:30.659034 24643 leader_election.cc:288] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:30.659051 24648 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:30.659173 24648 raft_consensus.cc:686] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95 [term 1 LEADER]: Becoming Leader. State: Replica: 298faf82f0c644ed87943af77f6dfc95, State: Running, Role: LEADER
I20260620 11:37:30.659307 24643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:30.659581 24648 consensus_queue.cc:237] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298faf82f0c644ed87943af77f6dfc95" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36871 } }
I20260620 11:37:30.659932 24649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "298faf82f0c644ed87943af77f6dfc95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298faf82f0c644ed87943af77f6dfc95" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36871 } } }
I20260620 11:37:30.660020 24649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:30.660250 24657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:30.660769 24657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 11:37:30 dist-test-slave-ctbw krb5kdc[24562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:37:30.661645 24650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 298faf82f0c644ed87943af77f6dfc95. Latest consensus state: current_term: 1 leader_uuid: "298faf82f0c644ed87943af77f6dfc95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298faf82f0c644ed87943af77f6dfc95" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36871 } } }
I20260620 11:37:30.661732 24650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:30.662159 24657 catalog_manager.cc:1347] Generated new cluster ID: 4b1a30297fea429dbc4bb030e3440ebc
I20260620 11:37:30.662211 24657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:37:30.665839 24646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:30.654532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47185 (local address 127.2.243.190:36871)
0620 11:37:30.654726 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:30.654739 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:30.654769 (+ 30us) server_negotiation.cc:407] Connection header received
0620 11:37:30.655443 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:30.655452 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:30.655505 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:30.655680 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:30.656471 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:30.657321 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:30.658381 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:30.658816 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:30.658950 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:30.662234 (+ 3284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:30.662238 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:30.662238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:30.662249 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:30.664188 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:30.664617 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:30.664620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:30.664620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:30.664721 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:30.664953 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:30.664954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:30.664954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:30.665252 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:30.665398 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:30.665521 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 11:37:30.665623 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":84,"threads_started":1}
I20260620 11:37:30.666811 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:37:30.673285 24657 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:30.673854 24657 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:30.680541 24646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:30.673214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47339 (local address 127.2.243.190:36871)
0620 11:37:30.673245 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:30.673247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:30.673493 (+ 246us) server_negotiation.cc:407] Connection header received
0620 11:37:30.673543 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:30.673544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:30.673554 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:30.673613 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:30.674182 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:30.674898 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:30.675899 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:30.676226 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:30.676290 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:30.677653 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:30.677654 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:30.677654 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:30.677660 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:30.679235 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:30.679658 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:30.679659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:30.679659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:30.679725 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:30.679941 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:30.679942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:30.679942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:30.680054 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:30.680143 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:30.680300 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 11:37:30.680366 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 11:37:30.680752 24657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 298faf82f0c644ed87943af77f6dfc95: Generated new TSK 0
I20260620 11:37:30.680900 24657 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:37:30.686142 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 24575
W20260620 11:37:30.690557 24560 connection.cc:535] client connection to 127.2.243.190:36871 recv error: Network error: failed to read from TLS socket (remote: 127.2.243.190:36871): Connection reset by peer (error 104)
2026-06-20T11:37:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:30 dist-test-slave-ctbw krb5kdc[24674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:30 dist-test-slave-ctbw krb5kdc[24674](info): set up 2 sockets
Jun 20 11:37:30 dist-test-slave-ctbw krb5kdc[24674](info): commencing operation
krb5kdc: starting...
W20260620 11:37:31.723098 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:31 dist-test-slave-ctbw krb5kdc[24674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:37:31.834712 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:38197
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:45175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:38197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:31.917073 24687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:31.917265 24687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:31.917287 24687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:31.918416 24687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:31.918458 24687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:31.918473 24687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:31.918524 24687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:31.918545 24687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:31.920037 24687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:45175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:38197
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:38197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:31.920288 24687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:31.920521 24687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:31.925380 24695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:31.925429 24692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:31.925393 24693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:31.925565 24687 server_base.cc:1034] running on GCE node
I20260620 11:37:31.925853 24687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:31.926173 24687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:31.927330 24687 hybrid_clock.cc:648] HybridClock initialized: now 1781955451927317 us; error 47 us; skew 500 ppm
Jun 20 11:37:31 dist-test-slave-ctbw krb5kdc[24674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955451, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:31.929656 24687 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:37:31.930068 24687 webserver.cc:460] Webserver started at http://127.2.243.190:39715/ using document root <none> and password file <none>
I20260620 11:37:31.930295 24687 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:31.930364 24687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:31.930510 24687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:31.931622 24687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "d2c07ac14a714cd9a17e7b4bf6a650eb"
format_stamp: "Formatted at 2026-06-20 11:37:31 on dist-test-slave-ctbw"
server_key: "1cbfa1cb83559617a3ff2d96f84e6a01"
server_key_iv: "4268eac2365b4889a7d1cd7fd0c10e16"
server_key_version: "encryptionkey@0"
I20260620 11:37:31.931968 24687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "d2c07ac14a714cd9a17e7b4bf6a650eb"
format_stamp: "Formatted at 2026-06-20 11:37:31 on dist-test-slave-ctbw"
server_key: "1cbfa1cb83559617a3ff2d96f84e6a01"
server_key_iv: "4268eac2365b4889a7d1cd7fd0c10e16"
server_key_version: "encryptionkey@0"
I20260620 11:37:31.933252 24687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:37:31.933928 24702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:31.934088 24687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:37:31.934190 24687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "d2c07ac14a714cd9a17e7b4bf6a650eb"
format_stamp: "Formatted at 2026-06-20 11:37:31 on dist-test-slave-ctbw"
server_key: "1cbfa1cb83559617a3ff2d96f84e6a01"
server_key_iv: "4268eac2365b4889a7d1cd7fd0c10e16"
server_key_version: "encryptionkey@0"
I20260620 11:37:31.934273 24687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:31.948318 24687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:31.950030 24687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:31.950186 24687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:31.953966 24687 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:38197
I20260620 11:37:31.954007 24754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:38197 every 8 connection(s)
I20260620 11:37:31.954360 24687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:31.955263 24755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:31.957386 24755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb: Bootstrap starting.
I20260620 11:37:31.957963 24755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:31.958243 24755 log.cc:824] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:31.958747 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 24687
I20260620 11:37:31.958899 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:31.959031 24755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb: No bootstrap required, opened a new log
I20260620 11:37:31.959105 3022 external_mini_cluster.cc:1428] Setting key 36958be1a97fbc3d89d507bcd264402b
I20260620 11:37:31.961550 24755 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c07ac14a714cd9a17e7b4bf6a650eb" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 38197 } }
I20260620 11:37:31.961737 24755 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:31.961776 24755 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2c07ac14a714cd9a17e7b4bf6a650eb, State: Initialized, Role: FOLLOWER
I20260620 11:37:31.961895 24755 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c07ac14a714cd9a17e7b4bf6a650eb" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 38197 } }
I20260620 11:37:31.961969 24755 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:31.961999 24755 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:31.962044 24755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:31.962704 24755 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c07ac14a714cd9a17e7b4bf6a650eb" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 38197 } }
I20260620 11:37:31.962831 24755 leader_election.cc:302] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2c07ac14a714cd9a17e7b4bf6a650eb; no voters:
I20260620 11:37:31.963043 24755 leader_election.cc:288] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:31.963063 24760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:31.963181 24760 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb [term 1 LEADER]: Becoming Leader. State: Replica: d2c07ac14a714cd9a17e7b4bf6a650eb, State: Running, Role: LEADER
I20260620 11:37:31.963304 24755 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:31.963313 24760 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c07ac14a714cd9a17e7b4bf6a650eb" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 38197 } }
I20260620 11:37:31.963757 24761 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2c07ac14a714cd9a17e7b4bf6a650eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c07ac14a714cd9a17e7b4bf6a650eb" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 38197 } } }
I20260620 11:37:31.964004 24761 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:31.963927 24762 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2c07ac14a714cd9a17e7b4bf6a650eb. Latest consensus state: current_term: 1 leader_uuid: "d2c07ac14a714cd9a17e7b4bf6a650eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2c07ac14a714cd9a17e7b4bf6a650eb" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 38197 } } }
I20260620 11:37:31.964255 24762 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:31.964665 24769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:31.964835 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:31.959605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59407 (local address 127.2.243.190:38197)
0620 11:37:31.959788 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:31.959804 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:31.959828 (+ 24us) server_negotiation.cc:407] Connection header received
0620 11:37:31.960319 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:31.960325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:31.960368 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:31.960475 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:31.960705 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:31.961540 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:31.962651 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:31.963257 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:31.963651 (+ 394us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:31.964051 (+ 400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:31.964054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:31.964054 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:31.964249 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:31.964284 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:31.964298 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:31.964380 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:31.964464 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 11:37:31.964585 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260620 11:37:31.965344 24769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:31.965783 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:37:31.966948 24769 catalog_manager.cc:1347] Generated new cluster ID: 583401e979d140da8e233d86eaa23786
I20260620 11:37:31.966993 24769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:37:31.973898 24759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:31.967293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35773 (local address 127.2.243.190:38197)
0620 11:37:31.967329 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:31.967331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:31.971096 (+ 3765us) server_negotiation.cc:407] Connection header received
0620 11:37:31.971158 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:31.971158 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:31.971168 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:31.971222 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:31.971320 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:31.972008 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:31.972728 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:31.972943 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:31.973026 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:31.973673 (+ 647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:31.973674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:31.973674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:31.973679 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:31.973692 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:31.973696 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:31.973746 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:31.973803 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 11:37:31.973811 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:37:31.984441 24769 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:31.984935 24769 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:31.995483 24769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2c07ac14a714cd9a17e7b4bf6a650eb: Generated new TSK 0
I20260620 11:37:31.995674 24769 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:37:32.002254 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 24687
W20260620 11:37:32.006559 24672 connection.cc:535] client connection to 127.2.243.190:38197 recv error: Network error: failed to read from TLS socket (remote: 127.2.243.190:38197): Connection reset by peer (error 104)
2026-06-20T11:37:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:32 dist-test-slave-ctbw krb5kdc[24786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:32 dist-test-slave-ctbw krb5kdc[24786](info): set up 2 sockets
Jun 20 11:37:32 dist-test-slave-ctbw krb5kdc[24786](info): commencing operation
krb5kdc: starting...
W20260620 11:37:33.023185 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.003s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:33 dist-test-slave-ctbw krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955453, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:37:33.134347 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:36533
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:36269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:36533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:33.214088 24799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:33.214334 24799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:33.214372 24799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:33.215543 24799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:33.215615 24799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:33.215637 24799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:33.215658 24799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:33.215687 24799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:33.217144 24799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:36269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:36533
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:36533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:33.217437 24799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:33.217685 24799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:33.222349 24805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:33.222389 24804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:33.222350 24807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:33.222679 24799 server_base.cc:1034] running on GCE node
I20260620 11:37:33.222932 24799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:33.223215 24799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:33.224366 24799 hybrid_clock.cc:648] HybridClock initialized: now 1781955453224351 us; error 37 us; skew 500 ppm
Jun 20 11:37:33 dist-test-slave-ctbw krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955453, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:33.229851 24799 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:37:33.230294 24799 webserver.cc:460] Webserver started at http://127.2.243.190:33471/ using document root <none> and password file <none>
I20260620 11:37:33.230489 24799 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:33.230530 24799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:33.230621 24799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:33.231675 24799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "55eebfcca2ac46018b1a433135c9a9ad"
format_stamp: "Formatted at 2026-06-20 11:37:33 on dist-test-slave-ctbw"
server_key: "1cb27d8579dea143177913c3759010cd"
server_key_iv: "b533edacdb426cf1c4656a20b9a1d8c3"
server_key_version: "encryptionkey@0"
I20260620 11:37:33.231972 24799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "55eebfcca2ac46018b1a433135c9a9ad"
format_stamp: "Formatted at 2026-06-20 11:37:33 on dist-test-slave-ctbw"
server_key: "1cb27d8579dea143177913c3759010cd"
server_key_iv: "b533edacdb426cf1c4656a20b9a1d8c3"
server_key_version: "encryptionkey@0"
I20260620 11:37:33.233240 24799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:37:33.234006 24814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:33.234182 24799 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:37:33.234278 24799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "55eebfcca2ac46018b1a433135c9a9ad"
format_stamp: "Formatted at 2026-06-20 11:37:33 on dist-test-slave-ctbw"
server_key: "1cb27d8579dea143177913c3759010cd"
server_key_iv: "b533edacdb426cf1c4656a20b9a1d8c3"
server_key_version: "encryptionkey@0"
I20260620 11:37:33.234337 24799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:33.237847 24799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:33.239861 24799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:33.239996 24799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:33.244036 24799 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:36533
I20260620 11:37:33.244086 24866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:36533 every 8 connection(s)
I20260620 11:37:33.244395 24799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:33.245380 24867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:33.247583 24867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad: Bootstrap starting.
I20260620 11:37:33.248129 24867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:33.248400 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 24799
I20260620 11:37:33.248404 24867 log.cc:824] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:33.248539 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:33.248776 3022 external_mini_cluster.cc:1428] Setting key 369857af53f48b693d5339e95fba3ae7
I20260620 11:37:33.249179 24867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad: No bootstrap required, opened a new log
I20260620 11:37:33.250828 24867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55eebfcca2ac46018b1a433135c9a9ad" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36533 } }
I20260620 11:37:33.251051 24867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:33.251134 24867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55eebfcca2ac46018b1a433135c9a9ad, State: Initialized, Role: FOLLOWER
I20260620 11:37:33.251600 24867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55eebfcca2ac46018b1a433135c9a9ad" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36533 } }
I20260620 11:37:33.251713 24867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:33.251768 24867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:33.251852 24867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:33.252153 24871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:33.249250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50523 (local address 127.2.243.190:36533)
0620 11:37:33.249431 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:33.249438 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:33.249460 (+ 22us) server_negotiation.cc:407] Connection header received
0620 11:37:33.250816 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:33.250823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:33.250847 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:33.251022 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:33.251678 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:33.251681 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:33.251682 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:33.251685 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:33.251739 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:33.251756 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:33.251830 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:33.251913 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 11:37:33.251955 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260620 11:37:33.252940 24867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55eebfcca2ac46018b1a433135c9a9ad" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36533 } }
I20260620 11:37:33.253054 24867 leader_election.cc:302] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55eebfcca2ac46018b1a433135c9a9ad; no voters:
I20260620 11:37:33.253104 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:37:33.253211 24867 leader_election.cc:288] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:33.253237 24872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:33.253343 24872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad [term 1 LEADER]: Becoming Leader. State: Replica: 55eebfcca2ac46018b1a433135c9a9ad, State: Running, Role: LEADER
I20260620 11:37:33.253453 24867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:33.253481 24872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55eebfcca2ac46018b1a433135c9a9ad" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36533 } }
I20260620 11:37:33.253865 24874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55eebfcca2ac46018b1a433135c9a9ad. Latest consensus state: current_term: 1 leader_uuid: "55eebfcca2ac46018b1a433135c9a9ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55eebfcca2ac46018b1a433135c9a9ad" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36533 } } }
I20260620 11:37:33.253950 24874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:33.254153 24873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55eebfcca2ac46018b1a433135c9a9ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55eebfcca2ac46018b1a433135c9a9ad" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36533 } } }
I20260620 11:37:33.254557 24873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:33.254745 24878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:33.255311 24878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:33.256273 24871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:33.254629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51675 (local address 127.2.243.190:36533)
0620 11:37:33.254664 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:33.254666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:33.255328 (+ 662us) server_negotiation.cc:407] Connection header received
0620 11:37:33.255372 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:33.255373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:33.255380 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:33.255424 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:33.256062 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:33.256063 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:33.256063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:33.256067 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:33.256082 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:33.256086 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:33.256145 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:33.256204 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 11:37:33.256218 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 11:37:33.256891 24878 catalog_manager.cc:1347] Generated new cluster ID: 3c253904e3a24d24bbbe9f30c56ecddb
I20260620 11:37:33.256938 24878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:37:33.264997 24878 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:33.265141 24878 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:33.272183 24878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55eebfcca2ac46018b1a433135c9a9ad: Generated new TSK 0
I20260620 11:37:33.272368 24878 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:37:33.278842 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 24799
2026-06-20T11:37:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1276 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:33 dist-test-slave-ctbw krb5kdc[24898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:33 dist-test-slave-ctbw krb5kdc[24898](info): set up 2 sockets
Jun 20 11:37:33 dist-test-slave-ctbw krb5kdc[24898](info): commencing operation
krb5kdc: starting...
W20260620 11:37:34.315184 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:34 dist-test-slave-ctbw krb5kdc[24898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:37:34.428829 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:41947
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:45315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:41947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:34.507189 24911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:34.507378 24911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:34.507404 24911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:34.508526 24911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:34.508569 24911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:34.508582 24911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:34.508595 24911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:34.508607 24911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:34.510039 24911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:45315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:41947
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:41947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:34.510280 24911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:34.510494 24911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:34.515079 24919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:34.515154 24911 server_base.cc:1034] running on GCE node
W20260620 11:37:34.515061 24917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:34.515061 24916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:34.515496 24911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:34.515728 24911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:34.516872 24911 hybrid_clock.cc:648] HybridClock initialized: now 1781955454516861 us; error 40 us; skew 500 ppm
Jun 20 11:37:34 dist-test-slave-ctbw krb5kdc[24898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955454, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:34.519363 24911 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:37:34.519714 24911 webserver.cc:460] Webserver started at http://127.2.243.190:46535/ using document root <none> and password file <none>
I20260620 11:37:34.519899 24911 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:34.519938 24911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:34.520023 24911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:34.521023 24911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "1ce021caff0c4f0485fbc03e693d0be7"
format_stamp: "Formatted at 2026-06-20 11:37:34 on dist-test-slave-ctbw"
server_key: "f4b14f8aab4a97c7735fdb34bcf4a5c3"
server_key_iv: "b4778bc9af4ae4ba8da46414afe6e9ed"
server_key_version: "encryptionkey@0"
I20260620 11:37:34.521308 24911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "1ce021caff0c4f0485fbc03e693d0be7"
format_stamp: "Formatted at 2026-06-20 11:37:34 on dist-test-slave-ctbw"
server_key: "f4b14f8aab4a97c7735fdb34bcf4a5c3"
server_key_iv: "b4778bc9af4ae4ba8da46414afe6e9ed"
server_key_version: "encryptionkey@0"
I20260620 11:37:34.522537 24911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 11:37:34.523269 24926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:34.523505 24911 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 11:37:34.523629 24911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "1ce021caff0c4f0485fbc03e693d0be7"
format_stamp: "Formatted at 2026-06-20 11:37:34 on dist-test-slave-ctbw"
server_key: "f4b14f8aab4a97c7735fdb34bcf4a5c3"
server_key_iv: "b4778bc9af4ae4ba8da46414afe6e9ed"
server_key_version: "encryptionkey@0"
I20260620 11:37:34.523712 24911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:34.545334 24911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:34.547885 24911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:34.548066 24911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:34.552327 24911 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:41947
I20260620 11:37:34.552382 24978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:41947 every 8 connection(s)
I20260620 11:37:34.552847 24911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:34.553027 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 24911
I20260620 11:37:34.553148 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:34.553375 3022 external_mini_cluster.cc:1428] Setting key de9b65a08160bded5975f11e96de8fe9
I20260620 11:37:34.553969 24979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:34.557394 24979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7: Bootstrap starting.
I20260620 11:37:34.558073 24979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:34.558395 24979 log.cc:824] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:34.559092 24979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7: No bootstrap required, opened a new log
Jun 20 11:37:34 dist-test-slave-ctbw krb5kdc[24898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:37:34.561537 24979 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce021caff0c4f0485fbc03e693d0be7" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41947 } }
I20260620 11:37:34.561702 24979 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:34.561757 24979 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ce021caff0c4f0485fbc03e693d0be7, State: Initialized, Role: FOLLOWER
I20260620 11:37:34.561902 24979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce021caff0c4f0485fbc03e693d0be7" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41947 } }
I20260620 11:37:34.561998 24979 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:34.562054 24979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:34.562105 24979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:34.562609 24979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce021caff0c4f0485fbc03e693d0be7" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41947 } }
I20260620 11:37:34.562721 24979 leader_election.cc:302] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ce021caff0c4f0485fbc03e693d0be7; no voters:
I20260620 11:37:34.562906 24979 leader_election.cc:288] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:34.562986 24984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:34.563146 24979 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:34.563210 24984 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7 [term 1 LEADER]: Becoming Leader. State: Replica: 1ce021caff0c4f0485fbc03e693d0be7, State: Running, Role: LEADER
I20260620 11:37:34.563336 24984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce021caff0c4f0485fbc03e693d0be7" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41947 } }
I20260620 11:37:34.563709 24984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ce021caff0c4f0485fbc03e693d0be7. Latest consensus state: current_term: 1 leader_uuid: "1ce021caff0c4f0485fbc03e693d0be7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce021caff0c4f0485fbc03e693d0be7" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41947 } } }
I20260620 11:37:34.563725 24985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ce021caff0c4f0485fbc03e693d0be7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ce021caff0c4f0485fbc03e693d0be7" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 41947 } } }
I20260620 11:37:34.563997 24984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:34.564028 24985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:34.564239 24994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:34.564962 24994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:34.565644 24982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:34.553872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59867 (local address 127.2.243.190:41947)
0620 11:37:34.554092 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:34.554102 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:34.554136 (+ 34us) server_negotiation.cc:407] Connection header received
0620 11:37:34.554744 (+ 608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:34.554765 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:34.554835 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:34.555025 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:34.555603 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:34.557125 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:34.557919 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:34.558601 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:34.558724 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:34.561400 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:34.561404 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:34.561404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:34.561417 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:34.563692 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:34.564174 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:34.564178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:34.564179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:34.564311 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:34.564587 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:34.564589 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:34.564589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:34.564895 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:34.565035 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:34.565272 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 11:37:34.565425 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":102,"threads_started":1}
I20260620 11:37:34.566654 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:37:34.567852 24994 catalog_manager.cc:1347] Generated new cluster ID: 671020e056924b73a12b47c432e32fe5
I20260620 11:37:34.567905 24994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:37:34.575516 24982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:34.568138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54709 (local address 127.2.243.190:41947)
0620 11:37:34.568175 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:34.568177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:34.568299 (+ 122us) server_negotiation.cc:407] Connection header received
0620 11:37:34.568352 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:34.568353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:34.568362 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:34.568419 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:34.569012 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:34.569814 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:34.570851 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:34.571242 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:34.571324 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:34.572666 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:34.572667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:34.572668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:34.572674 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:34.574289 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:34.574670 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:34.574671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:34.574671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:34.574727 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:34.574994 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:34.574995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:34.574995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:34.575120 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:34.575213 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:34.575355 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 11:37:34.575429 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:37:34.583017 24994 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:34.583544 24994 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:34.588112 24994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ce021caff0c4f0485fbc03e693d0be7: Generated new TSK 0
I20260620 11:37:34.588315 24994 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:37:34.594240 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 24911
2026-06-20T11:37:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1315 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:34 dist-test-slave-ctbw krb5kdc[25010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:34 dist-test-slave-ctbw krb5kdc[25010](info): set up 2 sockets
Jun 20 11:37:34 dist-test-slave-ctbw krb5kdc[25010](info): commencing operation
krb5kdc: starting...
W20260620 11:37:35.728009 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:35 dist-test-slave-ctbw krb5kdc[25010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:37:35.839881 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:39131
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:42897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:39131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:35.921859 25023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:35.922086 25023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:35.922123 25023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:35.923362 25023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:35.923421 25023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:35.923444 25023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:35.923472 25023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:35.923501 25023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:35.925063 25023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:42897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:39131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:39131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:35.925326 25023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:35.925552 25023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:35.930737 25028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:35.930740 25031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:35.931052 25023 server_base.cc:1034] running on GCE node
W20260620 11:37:35.930836 25029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:35.931334 25023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:35.931627 25023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:35.932796 25023 hybrid_clock.cc:648] HybridClock initialized: now 1781955455932777 us; error 35 us; skew 500 ppm
Jun 20 11:37:35 dist-test-slave-ctbw krb5kdc[25010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955455, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:35.935452 25023 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:37:35.935878 25023 webserver.cc:460] Webserver started at http://127.2.243.190:37537/ using document root <none> and password file <none>
I20260620 11:37:35.936123 25023 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:35.936185 25023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:35.936360 25023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:35.937371 25023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "4bb510949cc34673ab6568143a25a290"
format_stamp: "Formatted at 2026-06-20 11:37:35 on dist-test-slave-ctbw"
server_key: "554e08af635ac0d85760467efbc36a31"
server_key_iv: "d2a92a9943414555ab0a8cc9e6dc051c"
server_key_version: "encryptionkey@0"
I20260620 11:37:35.937708 25023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "4bb510949cc34673ab6568143a25a290"
format_stamp: "Formatted at 2026-06-20 11:37:35 on dist-test-slave-ctbw"
server_key: "554e08af635ac0d85760467efbc36a31"
server_key_iv: "d2a92a9943414555ab0a8cc9e6dc051c"
server_key_version: "encryptionkey@0"
I20260620 11:37:35.938988 25023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 11:37:35.939692 25038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:35.939899 25023 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:37:35.939987 25023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "4bb510949cc34673ab6568143a25a290"
format_stamp: "Formatted at 2026-06-20 11:37:35 on dist-test-slave-ctbw"
server_key: "554e08af635ac0d85760467efbc36a31"
server_key_iv: "d2a92a9943414555ab0a8cc9e6dc051c"
server_key_version: "encryptionkey@0"
I20260620 11:37:35.940060 25023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:35.950502 25023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:35.953136 25023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:35.953294 25023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:35.957398 25023 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:39131
I20260620 11:37:35.957463 25090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:39131 every 8 connection(s)
I20260620 11:37:35.957792 25023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:35.958647 25091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:35.960840 25091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290: Bootstrap starting.
I20260620 11:37:35.961396 25091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:35.961683 25091 log.cc:824] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:35.962359 25091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290: No bootstrap required, opened a new log
I20260620 11:37:35.963837 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 25023
I20260620 11:37:35.963969 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:35.964159 3022 external_mini_cluster.cc:1428] Setting key 7f6422854970eaf27d4a6c54d1e9401b
I20260620 11:37:35.964111 25091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb510949cc34673ab6568143a25a290" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39131 } }
I20260620 11:37:35.964272 25091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:35.964321 25091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb510949cc34673ab6568143a25a290, State: Initialized, Role: FOLLOWER
I20260620 11:37:35.964463 25091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb510949cc34673ab6568143a25a290" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39131 } }
I20260620 11:37:35.964562 25091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:35.964633 25091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:35.964738 25091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:35.965534 25091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb510949cc34673ab6568143a25a290" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39131 } }
I20260620 11:37:35.965620 25091 leader_election.cc:302] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bb510949cc34673ab6568143a25a290; no voters:
I20260620 11:37:35.965747 25091 leader_election.cc:288] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:35.965806 25096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:35.965963 25096 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290 [term 1 LEADER]: Becoming Leader. State: Replica: 4bb510949cc34673ab6568143a25a290, State: Running, Role: LEADER
I20260620 11:37:35.966013 25091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:35.966051 25096 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb510949cc34673ab6568143a25a290" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39131 } }
I20260620 11:37:35.966413 25098 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bb510949cc34673ab6568143a25a290. Latest consensus state: current_term: 1 leader_uuid: "4bb510949cc34673ab6568143a25a290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb510949cc34673ab6568143a25a290" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39131 } } }
I20260620 11:37:35.966410 25097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bb510949cc34673ab6568143a25a290" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb510949cc34673ab6568143a25a290" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39131 } } }
I20260620 11:37:35.966506 25098 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:35.966514 25097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:35.966785 25105 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:35.967394 25105 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:35.969472 25105 catalog_manager.cc:1347] Generated new cluster ID: 995a50a0d33a4b1690c4ebf8ab4c461d
I20260620 11:37:35.969528 25105 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:37:35.972566 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:35.964695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39003 (local address 127.2.243.190:39131)
0620 11:37:35.967464 (+ 2769us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:35.967473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:35.967508 (+ 35us) server_negotiation.cc:407] Connection header received
0620 11:37:35.967682 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:35.967690 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:35.967759 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:35.967895 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:35.968135 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:35.969293 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:35.970424 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:35.971107 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:35.971223 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:35.971914 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:35.971918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:35.971918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:35.972111 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:35.972148 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:35.972165 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:35.972238 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:35.972300 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 11:37:35.972361 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2722,"thread_start_us":162,"threads_started":1}
I20260620 11:37:35.973788 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:37:35.974074 25105 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:35.974934 25105 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:35.977362 25105 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bb510949cc34673ab6568143a25a290: Generated new TSK 0
I20260620 11:37:35.977546 25105 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:37:35.982546 25095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:35.979223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53183 (local address 127.2.243.190:39131)
0620 11:37:35.979264 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:35.979266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:35.979337 (+ 71us) server_negotiation.cc:407] Connection header received
0620 11:37:35.979364 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:35.979364 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:35.979385 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:35.979450 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:35.979530 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:35.980293 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:35.981333 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:35.981647 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:35.981728 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:35.982307 (+ 579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:35.982308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:35.982309 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:35.982313 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:35.982339 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:35.982344 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:35.982401 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:35.982448 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 11:37:35.982473 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:37:35.984486 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 25023
2026-06-20T11:37:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:36 dist-test-slave-ctbw krb5kdc[25122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:36 dist-test-slave-ctbw krb5kdc[25122](info): set up 2 sockets
Jun 20 11:37:36 dist-test-slave-ctbw krb5kdc[25122](info): commencing operation
krb5kdc: starting...
W20260620 11:37:37.019207 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:37 dist-test-slave-ctbw krb5kdc[25122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:37:37.131062 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:36229
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:39113
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:36229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:37.212152 25135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:37.212383 25135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:37.212435 25135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:37.213603 25135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:37.213670 25135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:37.213694 25135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:37.213724 25135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:37.213752 25135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:37.215214 25135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:39113
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:36229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:36229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:37.215478 25135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:37.215706 25135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:37.220517 25140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:37.220557 25141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:37.220538 25143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:37.221048 25135 server_base.cc:1034] running on GCE node
I20260620 11:37:37.221313 25135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:37.221623 25135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:37.222779 25135 hybrid_clock.cc:648] HybridClock initialized: now 1781955457222773 us; error 42 us; skew 500 ppm
Jun 20 11:37:37 dist-test-slave-ctbw krb5kdc[25122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955457, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:37.225394 25135 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:37:37.225939 25135 webserver.cc:460] Webserver started at http://127.2.243.190:41125/ using document root <none> and password file <none>
I20260620 11:37:37.226207 25135 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:37.226280 25135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:37.226438 25135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:37.227591 25135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "ac718cf7dc904916a8f1906548fba8f0"
format_stamp: "Formatted at 2026-06-20 11:37:37 on dist-test-slave-ctbw"
server_key: "c0051666efebc2556fd0e123dc305340"
server_key_iv: "22b520eb3f1057f53ccd3748b9244bdd"
server_key_version: "encryptionkey@0"
I20260620 11:37:37.228057 25135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "ac718cf7dc904916a8f1906548fba8f0"
format_stamp: "Formatted at 2026-06-20 11:37:37 on dist-test-slave-ctbw"
server_key: "c0051666efebc2556fd0e123dc305340"
server_key_iv: "22b520eb3f1057f53ccd3748b9244bdd"
server_key_version: "encryptionkey@0"
I20260620 11:37:37.229439 25135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:37:37.230187 25150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:37.230355 25135 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:37:37.230471 25135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "ac718cf7dc904916a8f1906548fba8f0"
format_stamp: "Formatted at 2026-06-20 11:37:37 on dist-test-slave-ctbw"
server_key: "c0051666efebc2556fd0e123dc305340"
server_key_iv: "22b520eb3f1057f53ccd3748b9244bdd"
server_key_version: "encryptionkey@0"
I20260620 11:37:37.230556 25135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:37.233940 25135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:37.236411 25135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:37.236586 25135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:37.240839 25135 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:36229
I20260620 11:37:37.240865 25202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:36229 every 8 connection(s)
I20260620 11:37:37.241211 25135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:37.242159 25203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:37.244405 25203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0: Bootstrap starting.
I20260620 11:37:37.244948 25203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:37.245256 25203 log.cc:824] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:37.245498 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 25135
I20260620 11:37:37.245616 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:37.245810 3022 external_mini_cluster.cc:1428] Setting key ea2f3c4cc5c1e87f45facb09f61a796a
I20260620 11:37:37.246045 25203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0: No bootstrap required, opened a new log
I20260620 11:37:37.248339 25203 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac718cf7dc904916a8f1906548fba8f0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36229 } }
I20260620 11:37:37.248504 25203 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:37:37.248569 25203 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac718cf7dc904916a8f1906548fba8f0, State: Initialized, Role: FOLLOWER
I20260620 11:37:37.248694 25203 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac718cf7dc904916a8f1906548fba8f0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36229 } }
I20260620 11:37:37.248791 25203 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:37:37.248831 25203 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:37:37.248831 25207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:37.246345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35487 (local address 127.2.243.190:36229)
0620 11:37:37.246548 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:37.246557 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:37.246589 (+ 32us) server_negotiation.cc:407] Connection header received
0620 11:37:37.247450 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:37.247459 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:37.247492 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:37.247647 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:37.248235 (+ 588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:37.248239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:37.248239 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:37.248243 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:37.248287 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:37.248301 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:37.248372 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:37.248430 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 11:37:37.248470 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":89,"threads_started":1}
I20260620 11:37:37.248884 25203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:37.249586 25203 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac718cf7dc904916a8f1906548fba8f0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36229 } }
I20260620 11:37:37.249692 25203 leader_election.cc:302] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac718cf7dc904916a8f1906548fba8f0; no voters:
I20260620 11:37:37.249781 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:37:37.249852 25203 leader_election.cc:288] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:37:37.249869 25208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:37.249976 25208 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0 [term 1 LEADER]: Becoming Leader. State: Replica: ac718cf7dc904916a8f1906548fba8f0, State: Running, Role: LEADER
I20260620 11:37:37.250082 25208 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac718cf7dc904916a8f1906548fba8f0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36229 } }
I20260620 11:37:37.250116 25203 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:37.250411 25209 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac718cf7dc904916a8f1906548fba8f0. Latest consensus state: current_term: 1 leader_uuid: "ac718cf7dc904916a8f1906548fba8f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac718cf7dc904916a8f1906548fba8f0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36229 } } }
I20260620 11:37:37.250517 25209 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:37.250674 25210 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac718cf7dc904916a8f1906548fba8f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac718cf7dc904916a8f1906548fba8f0" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 36229 } } }
I20260620 11:37:37.250730 25213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:37.250800 25210 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:37.251334 25213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:37.253063 25213 catalog_manager.cc:1347] Generated new cluster ID: 60d47774c484405e8583c8efa39d0608
I20260620 11:37:37.253118 25213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:37:37.253741 25207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:37.251314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59491 (local address 127.2.243.190:36229)
0620 11:37:37.251348 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:37.251350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:37.252669 (+ 1319us) server_negotiation.cc:407] Connection header received
0620 11:37:37.252750 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:37.252751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:37.252757 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:37.252822 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:37.253525 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:37.253527 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:37.253527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:37:37.253532 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:37.253544 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:37:37.253548 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:37.253611 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:37.253666 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 11:37:37.253680 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:37:37.264942 25213 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:37.265105 25213 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:37.273205 25213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac718cf7dc904916a8f1906548fba8f0: Generated new TSK 0
I20260620 11:37:37.273365 25213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:37:37.279348 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 25135
2026-06-20T11:37:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1295 ms)
[----------] 18 tests from AuthTokenIssuingTest (23647 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:37 dist-test-slave-ctbw krb5kdc[25234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:37 dist-test-slave-ctbw krb5kdc[25234](info): set up 2 sockets
Jun 20 11:37:37 dist-test-slave-ctbw krb5kdc[25234](info): commencing operation
krb5kdc: starting...
W20260620 11:37:38.311305 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:38 dist-test-slave-ctbw krb5kdc[25234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:37:38.421738 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:39619
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:39599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:39619,127.2.243.189:33979,127.2.243.188:37111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:38.501307 25247 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 11:37:38.501510 25247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:38.501528 25247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:38.501549 25247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:38.502697 25247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:38.502736 25247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:38.502750 25247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:38.502763 25247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:38.502775 25247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:38.504236 25247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:39599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:39619,127.2.243.189:33979,127.2.243.188:37111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:39619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:38.504484 25247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:38.504699 25247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:38.509297 25252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:38.509344 25255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:38.509277 25253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:38.509613 25247 server_base.cc:1034] running on GCE node
I20260620 11:37:38.509845 25247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:38.510123 25247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:38.511278 25247 hybrid_clock.cc:648] HybridClock initialized: now 1781955458511279 us; error 57 us; skew 500 ppm
Jun 20 11:37:38 dist-test-slave-ctbw krb5kdc[25234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955458, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:38.517647 25247 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:37:38.518090 25247 webserver.cc:460] Webserver started at http://127.2.243.190:32945/ using document root <none> and password file <none>
I20260620 11:37:38.518275 25247 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:38.518314 25247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:38.518399 25247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:38.519451 25247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "9469561b3b804deba289bd5ee4b6d1e2"
format_stamp: "Formatted at 2026-06-20 11:37:38 on dist-test-slave-ctbw"
server_key: "c9cafab012f8417183de425eb129a953"
server_key_iv: "792cb5990f15e31468608fa50d46cd41"
server_key_version: "encryptionkey@0"
I20260620 11:37:38.519742 25247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "9469561b3b804deba289bd5ee4b6d1e2"
format_stamp: "Formatted at 2026-06-20 11:37:38 on dist-test-slave-ctbw"
server_key: "c9cafab012f8417183de425eb129a953"
server_key_iv: "792cb5990f15e31468608fa50d46cd41"
server_key_version: "encryptionkey@0"
I20260620 11:37:38.520956 25247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:37:38.521631 25262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:38.521827 25247 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:37:38.521921 25247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "9469561b3b804deba289bd5ee4b6d1e2"
format_stamp: "Formatted at 2026-06-20 11:37:38 on dist-test-slave-ctbw"
server_key: "c9cafab012f8417183de425eb129a953"
server_key_iv: "792cb5990f15e31468608fa50d46cd41"
server_key_version: "encryptionkey@0"
I20260620 11:37:38.521976 25247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:38.537146 25247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:38.539156 25247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:38.539263 25247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:38.542770 25247 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:39619
I20260620 11:37:38.542860 25314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:39619 every 8 connection(s)
I20260620 11:37:38.543166 25247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:38.544142 25315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:38.545585 25315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39619 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:37:38.546074 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 25247
I20260620 11:37:38.546196 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:38.546445 3022 external_mini_cluster.cc:1428] Setting key e3e0d09a38d26b5ba9f468749b038379
Jun 20 11:37:38 dist-test-slave-ctbw krb5kdc[25234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955458, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.2.243.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.189@KRBTEST.COM" created.
I20260620 11:37:38.556927 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.546217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:39619 (local address 127.0.0.1:51914)
0620 11:37:38.546409 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.546457 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:38.547185 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:38.547479 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:38.547482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:38.547841 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:38.548081 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:38.548086 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.549473 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.549474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:38.550649 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:38.550652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.551194 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.551195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:38.551291 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.551779 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:38.551790 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:38.553327 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:38.555492 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:38.555499 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:38.555500 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:38.555698 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:38.555921 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:38.555921 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:38.555921 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:38.556009 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:38.556417 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:38.556422 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:38.556473 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 11:37:38.556558 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 11:37:38.556670 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":89,"threads_started":1}
I20260620 11:37:38.556931 25318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.546413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51914 (local address 127.2.243.190:39619)
0620 11:37:38.546542 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:38.546547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:38.546586 (+ 39us) server_negotiation.cc:407] Connection header received
0620 11:37:38.547351 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:38.547354 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:38.547371 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:38.547461 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:38.548200 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.549373 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.550727 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.551088 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.551169 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.553434 (+ 2265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:38.553441 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:38.553441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:38.553461 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:38.555421 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.555774 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.555776 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.555776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.555859 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.556065 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.556066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.556066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.556319 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:38.556426 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:38.556640 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 11:37:38.556770 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
I20260620 11:37:38.557668 25315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:37:38.558025 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.557887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:33979 (local address 127.0.0.1:35684)
0620 11:37:38.557944 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.557995 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.189:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 11:37:38.558177 25266 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.243.189:33979: connect: Connection refused (error 111)
I20260620 11:37:38.558566 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.558468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:33979 (local address 127.0.0.1:35690)
0620 11:37:38.558520 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.558545 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.189:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 11:37:38.558670 25315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.189:33979: Network error: Client connection negotiation failed: client connection to 127.2.243.189:33979: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.189.keytab.
Entry for principal kudu/127.2.243.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.189.keytab.
I20260620 11:37:38.570590 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.189:33979
--webserver_interface=127.2.243.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.189.keytab
--principal=kudu/127.2.243.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:39599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:39619,127.2.243.189:33979,127.2.243.188:37111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:37:38.607904 25315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } attempt: 1
I20260620 11:37:38.608443 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.608291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:33979 (local address 127.0.0.1:35702)
0620 11:37:38.608364 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.608413 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.189:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 11:37:38.608861 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.608756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:33979 (local address 127.0.0.1:35704)
0620 11:37:38.608803 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.608829 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.189:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 11:37:38.608971 25315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.189:33979: Network error: Client connection negotiation failed: client connection to 127.2.243.189:33979: connect: Connection refused (error 111)
W20260620 11:37:38.649283 25322 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 11:37:38.649471 25322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:38.649488 25322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:38.649507 25322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:38.650545 25322 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:38.650575 25322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:38.650589 25322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:38.650601 25322 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:38.650612 25322 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:38.651998 25322 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:39599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:39619,127.2.243.189:33979,127.2.243.188:37111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.189.keytab
--principal=kudu/127.2.243.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.189:33979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.243.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:38.652171 25322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:38.652328 25322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:38.656809 25327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:38.656826 25328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:38.656857 25322 server_base.cc:1034] running on GCE node
W20260620 11:37:38.656827 25330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:38.657265 25322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:38.657593 25322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:38.658756 25322 hybrid_clock.cc:648] HybridClock initialized: now 1781955458658751 us; error 35 us; skew 500 ppm
Jun 20 11:37:38 dist-test-slave-ctbw krb5kdc[25234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955458, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:38.661275 25322 init.cc:377] Logged in from keytab as kudu/127.2.243.189@KRBTEST.COM (short username kudu)
I20260620 11:37:38.661823 25322 webserver.cc:460] Webserver started at http://127.2.243.189:40563/ using document root <none> and password file <none>
I20260620 11:37:38.662074 25322 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:38.662139 25322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:38.662267 25322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:38.663337 25322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-1/data/instance:
uuid: "62dd33fb6ed84132a65011fa70ed92ed"
format_stamp: "Formatted at 2026-06-20 11:37:38 on dist-test-slave-ctbw"
server_key: "0e5f02478ac85d401d9459749bda6979"
server_key_iv: "f52ca419305e92af5998f6792cc164c2"
server_key_version: "encryptionkey@0"
I20260620 11:37:38.663682 25322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-1/wal/instance:
uuid: "62dd33fb6ed84132a65011fa70ed92ed"
format_stamp: "Formatted at 2026-06-20 11:37:38 on dist-test-slave-ctbw"
server_key: "0e5f02478ac85d401d9459749bda6979"
server_key_iv: "f52ca419305e92af5998f6792cc164c2"
server_key_version: "encryptionkey@0"
I20260620 11:37:38.664975 25322 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:37:38.665808 25337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:38.665988 25322 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:37:38.666102 25322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-1/wal
uuid: "62dd33fb6ed84132a65011fa70ed92ed"
format_stamp: "Formatted at 2026-06-20 11:37:38 on dist-test-slave-ctbw"
server_key: "0e5f02478ac85d401d9459749bda6979"
server_key_iv: "f52ca419305e92af5998f6792cc164c2"
server_key_version: "encryptionkey@0"
I20260620 11:37:38.666188 25322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:38.677114 25315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } attempt: 2
I20260620 11:37:38.677636 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.677516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:33979 (local address 127.0.0.1:35720)
0620 11:37:38.677572 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.677602 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.189:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 11:37:38.677961 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.677889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:33979 (local address 127.0.0.1:35734)
0620 11:37:38.677921 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.677940 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.189:33979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260620 11:37:38.678057 25315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.189:33979: Network error: Client connection negotiation failed: client connection to 127.2.243.189:33979: connect: Connection refused (error 111)
I20260620 11:37:38.680670 25322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:38.682474 25322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:38.682587 25322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:38.686985 25322 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.189:33979
I20260620 11:37:38.687067 25389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.189:33979 every 8 connection(s)
I20260620 11:37:38.687393 25322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-1/data/info.pb
I20260620 11:37:38.688309 25390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:38.689782 25390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39619 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:37:38.694993 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 25322
Jun 20 11:37:38 dist-test-slave-ctbw krb5kdc[25234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955458, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.189@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:37:38.695107 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-1/wal/instance
I20260620 11:37:38.695294 3022 external_mini_cluster.cc:1428] Setting key 2475286da0e2776a37be735eb1f04353
I20260620 11:37:38.698746 25318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.690447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51918 (local address 127.2.243.190:39619)
0620 11:37:38.690518 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:38.690520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:38.690704 (+ 184us) server_negotiation.cc:407] Connection header received
0620 11:37:38.691229 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:38.691231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:38.691240 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:38.691283 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:38.691834 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.692296 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.693241 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.693424 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.693476 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.695615 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:38.695617 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:38.695617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:38.695623 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:38.697336 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.697741 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.697742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.697743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.697802 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.698124 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.698125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.698126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.698244 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:38.698309 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:38.698620 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 11:37:38.698686 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 11:37:38.699091 25392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.690423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:39619 (local address 127.0.0.1:51918)
0620 11:37:38.690580 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.690619 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:38.691125 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:38.691334 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:38.691337 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:38.691581 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:38.691764 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:38.691768 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.692373 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.692373 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:38.693169 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:38.693171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.693502 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.693503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:38.693618 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.694039 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:38.694047 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:38.695526 (+ 1479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:38.697429 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:38.697436 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:38.697437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:38.697650 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:38.697876 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:38.697877 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:38.697877 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:38.698017 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:38.698318 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:38.698324 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:38.698484 (+ 160us) client_negotiation.cc:769] Sending connection context
0620 11:37:38.698596 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 11:37:38.698767 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260620 11:37:38.699700 25390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.243.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.188@KRBTEST.COM" created.
Jun 20 11:37:38 dist-test-slave-ctbw krb5kdc[25234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955458, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.189@KRBTEST.COM for kudu/127.2.243.189@KRBTEST.COM
I20260620 11:37:38.709594 25394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.700038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35738 (local address 127.2.243.189:33979)
0620 11:37:38.700164 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:38.700168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:38.700235 (+ 67us) server_negotiation.cc:407] Connection header received
0620 11:37:38.700304 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:38.700305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:38.700315 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:38.700374 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:38.700669 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.701312 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.702373 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.702671 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.702722 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.706678 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:38.706687 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:38.706688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:38.706695 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:38.708330 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.708743 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.708745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.708746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.708795 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.708989 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.708990 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.708990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.709121 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:38.709182 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:38.709388 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 11:37:38.709501 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":35,"server-negotiator.queue_time_us":106,"thread_start_us":84,"threads_started":1}
I20260620 11:37:38.709609 25392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.699975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:33979 (local address 127.0.0.1:35738)
0620 11:37:38.700176 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.700187 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:38.700244 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:38.700380 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:38.700381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:38.700537 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:38.700594 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:38.700596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.701397 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.701398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:38.702281 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:38.702284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.702752 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.702753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:38.702837 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.703208 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:38.703212 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:38.706595 (+ 3383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:38.708426 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:38.708427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:38.708428 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:38.708660 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:38.708859 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:38.708860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:38.708861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:38.708929 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:38.709190 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:38.709191 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:38.709240 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 11:37:38.709320 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 11:37:38.709444 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260620 11:37:38.710479 25390 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:37:38.710819 25392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.710703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37111 (local address 127.0.0.1:52012)
0620 11:37:38.710750 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.710793 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 11:37:38.710978 25339 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37111: connect: Connection refused (error 111)
I20260620 11:37:38.711361 25392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.711271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37111 (local address 127.0.0.1:52020)
0620 11:37:38.711317 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.711340 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 11:37:38.711470 25390 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.188:37111: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37111: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.188.keytab.
Entry for principal kudu/127.2.243.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.188.keytab.
I20260620 11:37:38.719847 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.188:37111
--webserver_interface=127.2.243.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.188.keytab
--principal=kudu/127.2.243.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:39599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:39619,127.2.243.189:33979,127.2.243.188:37111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:37:38.760704 25390 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } attempt: 1
I20260620 11:37:38.761179 25392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.761047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37111 (local address 127.0.0.1:52022)
0620 11:37:38.761110 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.761150 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 11:37:38.761528 25392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.761434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37111 (local address 127.0.0.1:52026)
0620 11:37:38.761480 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.761504 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 11:37:38.761619 25390 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.188:37111: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37111: connect: Connection refused (error 111)
I20260620 11:37:38.769181 25315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } attempt: 3
Jun 20 11:37:38 dist-test-slave-ctbw krb5kdc[25234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955458, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for kudu/127.2.243.189@KRBTEST.COM
I20260620 11:37:38.776067 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.769464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:33979 (local address 127.0.0.1:35752)
0620 11:37:38.769514 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.769525 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:38.769597 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:38.769771 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:38.769772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:38.769933 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:38.769995 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:38.769997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.770666 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.770667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:38.771245 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:38.771248 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.771651 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.771652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:38.771742 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.772233 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:38.772237 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:38.773452 (+ 1215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:38.775056 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:38.775057 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:38.775057 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:38.775227 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:38.775438 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:38.775439 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:38.775439 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:38.775497 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:38.775745 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:38.775746 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:38.775791 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 11:37:38.775878 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 11:37:38.775974 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 11:37:38.776063 25394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.769528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35752 (local address 127.2.243.189:33979)
0620 11:37:38.769597 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:38.769599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:38.769609 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:37:38.769682 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:38.769683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:38.769694 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:38.769765 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:38.770077 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.770574 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.771341 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.771575 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.771653 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.773539 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:38.773541 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:38.773541 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:38.773548 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:38.774977 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.775311 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.775312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.775313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.775370 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.775567 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.775568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.775568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.775677 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:38.775752 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:38.775902 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 11:37:38.775982 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 11:37:38.776649 25315 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:37:38.776980 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.776882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37111 (local address 127.0.0.1:52028)
0620 11:37:38.776928 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.776954 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 11:37:38.777372 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.777291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37111 (local address 127.0.0.1:52036)
0620 11:37:38.777335 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.777358 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 11:37:38.777503 25315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.188:37111: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37111: connect: Connection refused (error 111)
W20260620 11:37:38.800806 25397 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 11:37:38.801028 25397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:38.801052 25397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:38.801074 25397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:38.802160 25397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:38.802193 25397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:38.802207 25397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:38.802220 25397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:38.802232 25397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:38.803750 25397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:39599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:39619,127.2.243.189:33979,127.2.243.188:37111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.188.keytab
--principal=kudu/127.2.243.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.188:37111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.243.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:38.803943 25397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:38.804137 25397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:38.808676 25402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:38.808724 25315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } attempt: 1
W20260620 11:37:38.808668 25403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:38.808670 25405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:38.809094 25397 server_base.cc:1034] running on GCE node
I20260620 11:37:38.809129 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.808971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37111 (local address 127.0.0.1:52050)
0620 11:37:38.809041 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.809084 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 11:37:38.809443 25397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:38.809494 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.809392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37111 (local address 127.0.0.1:52058)
0620 11:37:38.809450 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.809474 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 11:37:38.809582 25315 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.188:37111: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37111: connect: Connection refused (error 111)
I20260620 11:37:38.809787 25397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:38.810945 25397 hybrid_clock.cc:648] HybridClock initialized: now 1781955458810928 us; error 28 us; skew 500 ppm
Jun 20 11:37:38 dist-test-slave-ctbw krb5kdc[25234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955458, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:38.813513 25397 init.cc:377] Logged in from keytab as kudu/127.2.243.188@KRBTEST.COM (short username kudu)
I20260620 11:37:38.813865 25397 webserver.cc:460] Webserver started at http://127.2.243.188:33853/ using document root <none> and password file <none>
I20260620 11:37:38.814049 25397 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:38.814090 25397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:38.814177 25397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:38.815212 25397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-2/data/instance:
uuid: "4d0a1f19824648a2b94f71d3b138a3d1"
format_stamp: "Formatted at 2026-06-20 11:37:38 on dist-test-slave-ctbw"
server_key: "720903ea3395d0d82e55b453d052a7a0"
server_key_iv: "e188e194d7665cc274ddaecde2b62e24"
server_key_version: "encryptionkey@0"
I20260620 11:37:38.815514 25397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-2/wal/instance:
uuid: "4d0a1f19824648a2b94f71d3b138a3d1"
format_stamp: "Formatted at 2026-06-20 11:37:38 on dist-test-slave-ctbw"
server_key: "720903ea3395d0d82e55b453d052a7a0"
server_key_iv: "e188e194d7665cc274ddaecde2b62e24"
server_key_version: "encryptionkey@0"
I20260620 11:37:38.816717 25397 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:37:38.817353 25412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:38.817552 25397 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:37:38.817633 25397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-2/wal
uuid: "4d0a1f19824648a2b94f71d3b138a3d1"
format_stamp: "Formatted at 2026-06-20 11:37:38 on dist-test-slave-ctbw"
server_key: "720903ea3395d0d82e55b453d052a7a0"
server_key_iv: "e188e194d7665cc274ddaecde2b62e24"
server_key_version: "encryptionkey@0"
I20260620 11:37:38.817684 25397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:38.829756 25390 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } attempt: 2
I20260620 11:37:38.830125 25392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.830005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37111 (local address 127.0.0.1:52062)
0620 11:37:38.830073 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.830099 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 11:37:38.830502 25392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.830418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37111 (local address 127.0.0.1:52068)
0620 11:37:38.830461 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.830485 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 11:37:38.830600 25390 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.188:37111: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37111: connect: Connection refused (error 111)
I20260620 11:37:38.833498 25397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:38.835335 25397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:38.835486 25397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:38.839123 25397 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.188:37111
I20260620 11:37:38.839148 25464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.188:37111 every 8 connection(s)
I20260620 11:37:38.839485 25397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-2/data/info.pb
I20260620 11:37:38.840443 25465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:38.841902 25465 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39619 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:37:38.843935 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 25397
I20260620 11:37:38.844038 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955192827595-3022-0/minicluster-data/master-2/wal/instance
I20260620 11:37:38.844209 3022 external_mini_cluster.cc:1428] Setting key 582329c019bffaf2047f9e79fa788d8a
Jun 20 11:37:38 dist-test-slave-ctbw krb5kdc[25234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955458, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.188@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
Jun 20 11:37:38 dist-test-slave-ctbw krb5kdc[25234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.188@KRBTEST.COM
I20260620 11:37:38.851517 25318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.842561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51926 (local address 127.2.243.190:39619)
0620 11:37:38.842609 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:38.842611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:38.842813 (+ 202us) server_negotiation.cc:407] Connection header received
0620 11:37:38.843337 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:38.843338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:38.843348 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:38.843400 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:38.843986 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.844697 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.846033 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.846327 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.846402 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.848643 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:38.848645 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:38.848645 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:38.848653 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:38.850027 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.850499 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.850499 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.850500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.850550 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.850839 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.850840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.850840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.850992 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:38.851067 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:38.851349 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 11:37:38.851429 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:37:38.851691 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.842547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:39619 (local address 127.0.0.1:51926)
0620 11:37:38.842702 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.842736 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:38.843250 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:38.843462 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:38.843465 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:38.843688 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:38.843887 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:38.843891 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.844790 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.844791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:38.845953 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:38.845956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.846426 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.846427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:38.846516 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.847076 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:38.847086 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:38.848552 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:38.850114 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:38.850120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:38.850122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:38.850422 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:38.850630 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:38.850631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:38.850632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:38.850769 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:38.851070 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:38.851077 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:38.851206 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 11:37:38.851314 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 11:37:38.851466 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260620 11:37:38.852259 25465 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:37:38.853930 25469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.844566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52082 (local address 127.2.243.188:37111)
0620 11:37:38.844728 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:38.844732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:38.844799 (+ 67us) server_negotiation.cc:407] Connection header received
0620 11:37:38.844868 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:38.844869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:38.844886 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:38.844940 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:38.845675 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.846563 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.847598 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.847872 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.847976 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.850603 (+ 2627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:38.850610 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:38.850610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:38.850617 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:38.852606 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.852972 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.852975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.852975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.853042 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.853245 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.853246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.853247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.853430 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:38.853557 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:38.853756 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 11:37:38.853832 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":102,"threads_started":1}
Jun 20 11:37:38 dist-test-slave-ctbw krb5kdc[25234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955458, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.188@KRBTEST.COM for kudu/127.2.243.189@KRBTEST.COM
I20260620 11:37:38.858852 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.852507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:33979 (local address 127.0.0.1:35754)
0620 11:37:38.852547 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.852555 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:38.852587 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:38.852774 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:38.852775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:38.852911 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:38.852937 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:38.852938 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.853561 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.853562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:38.854271 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:38.854273 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.854760 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.854761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:38.854805 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.855213 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:38.855217 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:38.856356 (+ 1139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:38.857917 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:38.857918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:38.857918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:38.858140 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:38.858332 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:38.858333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:38.858333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:38.858399 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:38.858614 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:38.858615 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:38.858663 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 11:37:38.858735 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 11:37:38.858798 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 11:37:38.858994 25394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.852591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35754 (local address 127.2.243.189:33979)
0620 11:37:38.852659 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:38.852661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:38.852668 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:37:38.852691 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:38.852692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:38.852701 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:38.852771 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:38.853022 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.853487 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.854382 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.854703 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.854754 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.856449 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:38.856450 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:38.856450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:38.856456 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:38.857836 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.858222 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.858223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.858223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.858269 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.858459 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.858459 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.858459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.858548 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:38.858596 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:38.858776 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 11:37:38.858849 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 11:37:38 dist-test-slave-ctbw krb5kdc[25234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:37:38.859360 25465 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:37:38.862274 25318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.854650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51938 (local address 127.2.243.190:39619)
0620 11:37:38.854683 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:38.854685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:38.854900 (+ 215us) server_negotiation.cc:407] Connection header received
0620 11:37:38.854923 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:38.854924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:38.854934 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:38.854984 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:38.855496 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.855987 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.856831 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.857120 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.857195 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.859734 (+ 2539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:38.859735 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:38.859736 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:38.859743 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:38.861148 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.861480 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.861481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.861482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.861543 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.861739 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.861740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.861741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.861844 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:38.862026 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:38.862107 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 11:37:38.862180 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Jun 20 11:37:38 dist-test-slave-ctbw krb5kdc[25234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955458, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.188@KRBTEST.COM for kudu/127.2.243.188@KRBTEST.COM
I20260620 11:37:38.865828 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.859598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37111 (local address 127.0.0.1:52088)
0620 11:37:38.859636 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.859644 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:38.859676 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:38.859776 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:38.859777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:38.859938 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:38.859966 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:38.859968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.860871 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.860872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:38.861306 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:38.861308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.861699 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.861700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:38.861747 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.862124 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:38.862129 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:38.863502 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:38.864881 (+ 1379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:38.864882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:38.864882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:38.865043 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:38.865245 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:38.865245 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:38.865246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:38.865296 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:38.865539 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:38.865539 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:38.865575 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 11:37:38.865641 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 11:37:38.865729 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 11:37:38.865882 25469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.859651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52088 (local address 127.2.243.188:37111)
0620 11:37:38.859685 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:38.859687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:38.859695 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:37:38.859724 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:38.859725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:38.859733 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:38.859776 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:38.860072 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.860788 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.861413 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.861617 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.861698 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.863559 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:38.863559 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:38.863560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:38.863566 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:38.864785 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.865125 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.865126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.865126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.865172 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.865366 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.865368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.865368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.865474 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:38.865545 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:38.865667 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 11:37:38.865775 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":20}
Jun 20 11:37:38 dist-test-slave-ctbw krb5kdc[25234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955458, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.189@KRBTEST.COM
I20260620 11:37:38.867329 25465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d0a1f19824648a2b94f71d3b138a3d1: Bootstrap starting.
I20260620 11:37:38.868197 25465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d0a1f19824648a2b94f71d3b138a3d1: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:38.868535 25465 log.cc:824] T 00000000000000000000000000000000 P 4d0a1f19824648a2b94f71d3b138a3d1: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:38.869341 25465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d0a1f19824648a2b94f71d3b138a3d1: No bootstrap required, opened a new log
I20260620 11:37:38.870304 25394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.863040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35766 (local address 127.2.243.189:33979)
0620 11:37:38.863076 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:38.863078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:38.863326 (+ 248us) server_negotiation.cc:407] Connection header received
0620 11:37:38.863377 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:38.863378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:38.863389 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:38.863454 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:38.863926 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.864540 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.865159 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.865361 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.865444 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.867721 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:38.867722 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:38.867722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:38.867729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:38.869176 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.869547 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.869548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.869549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.869606 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.869805 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.869806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.869806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.869944 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:38.870090 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:38.870128 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 11:37:38.870194 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:37:38.871016 25465 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d0a1f19824648a2b94f71d3b138a3d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9469561b3b804deba289bd5ee4b6d1e2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39619 } } peers { permanent_uuid: "62dd33fb6ed84132a65011fa70ed92ed" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } } peers { permanent_uuid: "4d0a1f19824648a2b94f71d3b138a3d1" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } }
I20260620 11:37:38.871162 25465 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d0a1f19824648a2b94f71d3b138a3d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0a1f19824648a2b94f71d3b138a3d1, State: Initialized, Role: FOLLOWER
I20260620 11:37:38.871248 25465 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d0a1f19824648a2b94f71d3b138a3d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9469561b3b804deba289bd5ee4b6d1e2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39619 } } peers { permanent_uuid: "62dd33fb6ed84132a65011fa70ed92ed" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } } peers { permanent_uuid: "4d0a1f19824648a2b94f71d3b138a3d1" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } }
I20260620 11:37:38.871573 25465 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d0a1f19824648a2b94f71d3b138a3d1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:38.871658 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:37:38.871619 25471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d0a1f19824648a2b94f71d3b138a3d1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9469561b3b804deba289bd5ee4b6d1e2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39619 } } peers { permanent_uuid: "62dd33fb6ed84132a65011fa70ed92ed" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } } peers { permanent_uuid: "4d0a1f19824648a2b94f71d3b138a3d1" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } } }
I20260620 11:37:38.871716 25471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d0a1f19824648a2b94f71d3b138a3d1 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 11:37:38.873395 25482 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4d0a1f19824648a2b94f71d3b138a3d1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:37:38.873458 25482 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:37:38.872750 25289 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9469561b3b804deba289bd5ee4b6d1e2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51938
I20260620 11:37:38.873785 25397 master_runner.cc:186] Error getting master registration for 127.2.243.190:39619: OK, instance_id { permanent_uuid: "9469561b3b804deba289bd5ee4b6d1e2" instance_seqno: 1781955458541796 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 11:37:38.876571 25397 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 11:37:38.884790 25315 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } attempt: 2
Jun 20 11:37:38 dist-test-slave-ctbw krb5kdc[25234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955458, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for kudu/127.2.243.188@KRBTEST.COM
I20260620 11:37:38.892763 25317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.885091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37111 (local address 127.0.0.1:52102)
0620 11:37:38.885143 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.885155 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:38.885196 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:38.885341 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:38.885342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:38.885466 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:38.885503 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:38.885504 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.886393 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.886394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:38.887192 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:38.887195 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.887662 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.887663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:38.887718 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.888199 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:38.888205 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:38.889792 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:38.891642 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:38.891645 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:38.891645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:38.891893 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:38.892097 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:38.892098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:38.892099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:38.892159 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:38.892454 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:38.892455 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:38.892495 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 11:37:38.892570 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 11:37:38.892661 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 11:37:38.892773 25469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.885173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52102 (local address 127.2.243.188:37111)
0620 11:37:38.885226 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:38.885228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:38.885236 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:37:38.885258 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:38.885259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:38.885269 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:38.885332 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:38.885583 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.886311 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.887275 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.887583 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.887658 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.889879 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:38.889880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:38.889881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:38.889887 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:38.891537 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.891967 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.891968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.891969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.892028 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.892225 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.892226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.892226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.892385 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:38.892455 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:38.892609 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 11:37:38.892686 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:37:38.894373 25315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2: Bootstrap starting.
I20260620 11:37:38.895107 25315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:38.895401 25315 log.cc:824] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:38.896250 25315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2: No bootstrap required, opened a new log
I20260620 11:37:38.897622 25315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9469561b3b804deba289bd5ee4b6d1e2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39619 } } peers { permanent_uuid: "62dd33fb6ed84132a65011fa70ed92ed" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } } peers { permanent_uuid: "4d0a1f19824648a2b94f71d3b138a3d1" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } }
I20260620 11:37:38.897760 25315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9469561b3b804deba289bd5ee4b6d1e2, State: Initialized, Role: FOLLOWER
I20260620 11:37:38.897832 25315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9469561b3b804deba289bd5ee4b6d1e2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39619 } } peers { permanent_uuid: "62dd33fb6ed84132a65011fa70ed92ed" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } } peers { permanent_uuid: "4d0a1f19824648a2b94f71d3b138a3d1" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } }
I20260620 11:37:38.898135 25315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:38.898108 25485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9469561b3b804deba289bd5ee4b6d1e2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39619 } } peers { permanent_uuid: "62dd33fb6ed84132a65011fa70ed92ed" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } } peers { permanent_uuid: "4d0a1f19824648a2b94f71d3b138a3d1" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } } }
I20260620 11:37:38.898195 25485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:37:38.898200 25289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 11:37:38.898312 25289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:38.899114 25289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9469561b3b804deba289bd5ee4b6d1e2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39619 } } peers { permanent_uuid: "62dd33fb6ed84132a65011fa70ed92ed" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } } peers { permanent_uuid: "4d0a1f19824648a2b94f71d3b138a3d1" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } }
I20260620 11:37:38.899380 25289 leader_election.cc:288] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2 [CANDIDATE]: Term 1 election: Requested vote from peers 62dd33fb6ed84132a65011fa70ed92ed (127.2.243.189:33979), 4d0a1f19824648a2b94f71d3b138a3d1 (127.2.243.188:37111)
I20260620 11:37:38.899833 25439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9469561b3b804deba289bd5ee4b6d1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4d0a1f19824648a2b94f71d3b138a3d1"
I20260620 11:37:38.899977 25439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d0a1f19824648a2b94f71d3b138a3d1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:38.900941 25439 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4d0a1f19824648a2b94f71d3b138a3d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9469561b3b804deba289bd5ee4b6d1e2 in term 1.
I20260620 11:37:38.901314 25264 leader_election.cc:302] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0a1f19824648a2b94f71d3b138a3d1, 9469561b3b804deba289bd5ee4b6d1e2; no voters:
W20260620 11:37:38.900116 25496 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:37:38.901404 25496 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:37:38.901497 25485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:38.901640 25485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2 [term 1 LEADER]: Becoming Leader. State: Replica: 9469561b3b804deba289bd5ee4b6d1e2, State: Running, Role: LEADER
I20260620 11:37:38.901718 25485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9469561b3b804deba289bd5ee4b6d1e2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39619 } } peers { permanent_uuid: "62dd33fb6ed84132a65011fa70ed92ed" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } } peers { permanent_uuid: "4d0a1f19824648a2b94f71d3b138a3d1" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } }
I20260620 11:37:38.902496 25505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9469561b3b804deba289bd5ee4b6d1e2. Latest consensus state: current_term: 1 leader_uuid: "9469561b3b804deba289bd5ee4b6d1e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9469561b3b804deba289bd5ee4b6d1e2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39619 } } peers { permanent_uuid: "62dd33fb6ed84132a65011fa70ed92ed" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } } peers { permanent_uuid: "4d0a1f19824648a2b94f71d3b138a3d1" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } } }
I20260620 11:37:38.902592 25505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:38.902921 25247 master_runner.cc:186] Error getting master registration for 127.2.243.189:33979: OK, instance_id { permanent_uuid: "62dd33fb6ed84132a65011fa70ed92ed" instance_seqno: 1781955458685730 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 11:37:38.905172 25247 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 11:37:38.913064 25507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:38.913288 25507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:38.915664 25469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.901007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52110 (local address 127.2.243.188:37111)
0620 11:37:38.901039 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:38.901041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:38.907271 (+ 6230us) server_negotiation.cc:407] Connection header received
0620 11:37:38.907384 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:38.907385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:38.907395 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:38.907424 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:38.908123 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.908829 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.909924 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.910220 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.910262 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.911883 (+ 1621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:38.911884 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:38.911885 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:38.911891 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:38.913636 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.914269 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.914270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.914271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.914348 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.914695 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.914696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.914697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.914845 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:38.914942 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:38.915502 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 11:37:38.915572 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 11:37:38.921257 25318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.900697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51942 (local address 127.2.243.190:39619)
0620 11:37:38.900735 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:38.900737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:38.907059 (+ 6322us) server_negotiation.cc:407] Connection header received
0620 11:37:38.907180 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:38.907181 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:38.907210 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:38.907269 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:38.908384 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.909142 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.910637 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.911015 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.911056 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.917539 (+ 6483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:38.917552 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:38.917552 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:38.917559 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:38.919506 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.920125 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.920126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.920127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.920192 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.920479 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.920480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.920480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.920614 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:38.920695 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:38.921091 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 11:37:38.921167 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:37:38.921718 25390 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } attempt: 3
I20260620 11:37:38.922143 25394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.901281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35778 (local address 127.2.243.189:33979)
0620 11:37:38.901321 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:38.901324 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:38.912245 (+ 10921us) server_negotiation.cc:407] Connection header received
0620 11:37:38.912344 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:38.912345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:38.912355 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:38.912386 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:38.914016 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.914769 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.915799 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.916122 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.916162 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.917581 (+ 1419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:38.917582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:38.917582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:38.917589 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:38.920246 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.920700 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.920701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.920702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.920758 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.921105 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.921106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.921106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.921234 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:38.921733 (+ 499us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:38.921953 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 11:37:38.922055 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:37:38.923352 25439 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4d0a1f19824648a2b94f71d3b138a3d1 [term 1 FOLLOWER]: Refusing update from remote peer 9469561b3b804deba289bd5ee4b6d1e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:37:38.923727 25505 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0a1f19824648a2b94f71d3b138a3d1" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:37:38.925246 25471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d0a1f19824648a2b94f71d3b138a3d1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9469561b3b804deba289bd5ee4b6d1e2. Latest consensus state: current_term: 1 leader_uuid: "9469561b3b804deba289bd5ee4b6d1e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9469561b3b804deba289bd5ee4b6d1e2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39619 } } peers { permanent_uuid: "62dd33fb6ed84132a65011fa70ed92ed" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } } peers { permanent_uuid: "4d0a1f19824648a2b94f71d3b138a3d1" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } } }
I20260620 11:37:38.925355 25471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d0a1f19824648a2b94f71d3b138a3d1 [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 11:37:38 dist-test-slave-ctbw krb5kdc[25234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955458, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.189@KRBTEST.COM for kudu/127.2.243.188@KRBTEST.COM
I20260620 11:37:38.930630 25392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.921975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37111 (local address 127.0.0.1:52122)
0620 11:37:38.922073 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:38.922083 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:38.922121 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:38.922444 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:38.922445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:38.922593 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:38.922631 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:38.922632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.923723 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.923724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:38.924557 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:38.924560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.925429 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.925430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:38.925503 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.926004 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:38.926021 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:38.927536 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:38.929508 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:38.929509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:38.929509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:38.929712 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:38.929917 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:38.929918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:38.929919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:38.929990 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:38.930291 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:38.930292 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:38.930343 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 11:37:38.930401 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 11:37:38.930496 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260620 11:37:38.930975 25469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:38.922228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52122 (local address 127.2.243.188:37111)
0620 11:37:38.922312 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:38.922314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:38.922322 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:37:38.922341 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:38.922342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:38.922351 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:38.922405 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:38.922705 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.923442 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.924800 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:38.925118 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:38.925176 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:38.927613 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:38.927614 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:38.927614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:38.927622 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:38.929421 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.929778 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.929779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.929779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.929833 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:38.930073 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:38.930074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:38.930075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:38.930198 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:38.930253 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:38.930792 (+ 539us) server_negotiation.cc:299] Negotiation successful
0620 11:37:38.930890 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 11:37:38.931836 25505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9469561b3b804deba289bd5ee4b6d1e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9469561b3b804deba289bd5ee4b6d1e2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39619 } } peers { permanent_uuid: "62dd33fb6ed84132a65011fa70ed92ed" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } } peers { permanent_uuid: "4d0a1f19824648a2b94f71d3b138a3d1" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } } }
I20260620 11:37:38.931910 25505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:38.932186 25471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d0a1f19824648a2b94f71d3b138a3d1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9469561b3b804deba289bd5ee4b6d1e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9469561b3b804deba289bd5ee4b6d1e2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39619 } } peers { permanent_uuid: "62dd33fb6ed84132a65011fa70ed92ed" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } } peers { permanent_uuid: "4d0a1f19824648a2b94f71d3b138a3d1" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } } }
I20260620 11:37:38.932260 25471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d0a1f19824648a2b94f71d3b138a3d1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:37:38.932654 25390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62dd33fb6ed84132a65011fa70ed92ed: Bootstrap starting.
I20260620 11:37:38.932737 25509 mvcc.cc:205] Tried to move back new op lower bound from 7298889559739510784 to 7298889559662231552. Current Snapshot: MvccSnapshot[applied={T|T < 7298889559739510784}]
I20260620 11:37:38.933346 25390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62dd33fb6ed84132a65011fa70ed92ed: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:38.933629 25390 log.cc:824] T 00000000000000000000000000000000 P 62dd33fb6ed84132a65011fa70ed92ed: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:38.934474 25507 catalog_manager.cc:1347] Generated new cluster ID: 41c471dc217c456b9a1ba355ef46d755
I20260620 11:37:38.934526 25507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:37:38.939457 25390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62dd33fb6ed84132a65011fa70ed92ed: No bootstrap required, opened a new log
I20260620 11:37:38.941457 25390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62dd33fb6ed84132a65011fa70ed92ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9469561b3b804deba289bd5ee4b6d1e2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39619 } } peers { permanent_uuid: "62dd33fb6ed84132a65011fa70ed92ed" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } } peers { permanent_uuid: "4d0a1f19824648a2b94f71d3b138a3d1" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } }
I20260620 11:37:38.941578 25390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62dd33fb6ed84132a65011fa70ed92ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62dd33fb6ed84132a65011fa70ed92ed, State: Initialized, Role: FOLLOWER
I20260620 11:37:38.941650 25390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62dd33fb6ed84132a65011fa70ed92ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9469561b3b804deba289bd5ee4b6d1e2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39619 } } peers { permanent_uuid: "62dd33fb6ed84132a65011fa70ed92ed" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } } peers { permanent_uuid: "4d0a1f19824648a2b94f71d3b138a3d1" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } }
I20260620 11:37:38.941931 25390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62dd33fb6ed84132a65011fa70ed92ed [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:38.942981 25364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9469561b3b804deba289bd5ee4b6d1e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "62dd33fb6ed84132a65011fa70ed92ed"
I20260620 11:37:38.943096 25364 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62dd33fb6ed84132a65011fa70ed92ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:38.943881 25364 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 62dd33fb6ed84132a65011fa70ed92ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9469561b3b804deba289bd5ee4b6d1e2 in term 1.
I20260620 11:37:38.944276 25363 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 62dd33fb6ed84132a65011fa70ed92ed [term 1 FOLLOWER]: Refusing update from remote peer 9469561b3b804deba289bd5ee4b6d1e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:37:38.944563 25485 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62dd33fb6ed84132a65011fa70ed92ed" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:37:38.950074 25507 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:38.950790 25507 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:38.950959 25514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62dd33fb6ed84132a65011fa70ed92ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9469561b3b804deba289bd5ee4b6d1e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9469561b3b804deba289bd5ee4b6d1e2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39619 } } peers { permanent_uuid: "62dd33fb6ed84132a65011fa70ed92ed" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } } peers { permanent_uuid: "4d0a1f19824648a2b94f71d3b138a3d1" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } } }
I20260620 11:37:38.951056 25514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62dd33fb6ed84132a65011fa70ed92ed [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:37:38.951114 25514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62dd33fb6ed84132a65011fa70ed92ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9469561b3b804deba289bd5ee4b6d1e2. Latest consensus state: current_term: 1 leader_uuid: "9469561b3b804deba289bd5ee4b6d1e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9469561b3b804deba289bd5ee4b6d1e2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39619 } } peers { permanent_uuid: "62dd33fb6ed84132a65011fa70ed92ed" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } } peers { permanent_uuid: "4d0a1f19824648a2b94f71d3b138a3d1" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } } }
I20260620 11:37:38.951162 25514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62dd33fb6ed84132a65011fa70ed92ed [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:37:38.951198 25514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62dd33fb6ed84132a65011fa70ed92ed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9469561b3b804deba289bd5ee4b6d1e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9469561b3b804deba289bd5ee4b6d1e2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39619 } } peers { permanent_uuid: "62dd33fb6ed84132a65011fa70ed92ed" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } } peers { permanent_uuid: "4d0a1f19824648a2b94f71d3b138a3d1" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } } }
I20260620 11:37:38.951241 25514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62dd33fb6ed84132a65011fa70ed92ed [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:37:38.952003 25521 mvcc.cc:205] Tried to move back new op lower bound from 7298889559739510784 to 7298889559662231552. Current Snapshot: MvccSnapshot[applied={T|T < 7298889559739510784}]
I20260620 11:37:38.952723 25485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9469561b3b804deba289bd5ee4b6d1e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9469561b3b804deba289bd5ee4b6d1e2" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 39619 } } peers { permanent_uuid: "62dd33fb6ed84132a65011fa70ed92ed" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 33979 } } peers { permanent_uuid: "4d0a1f19824648a2b94f71d3b138a3d1" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37111 } } }
I20260620 11:37:38.952831 25485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:38.955794 25529 catalog_manager.cc:1259] Loaded cluster ID: 41c471dc217c456b9a1ba355ef46d755
I20260620 11:37:38.955843 25529 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 62dd33fb6ed84132a65011fa70ed92ed: loading cluster ID for follower catalog manager: success
I20260620 11:37:38.956923 25529 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 62dd33fb6ed84132a65011fa70ed92ed: acquiring CA information for follower catalog manager: success
W20260620 11:37:38.957057 25529 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260620 11:37:38.957723 25507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9469561b3b804deba289bd5ee4b6d1e2: Generated new TSK 0
I20260620 11:37:38.957916 25507 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 11:37:38.980440 25394 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:38.978877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35784 (local address 127.2.243.189:33979)
0620 11:37:38.978920 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:38.978922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:38.979783 (+ 861us) server_negotiation.cc:407] Connection header received
0620 11:37:38.979886 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:38.979887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:38.979898 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:38.979929 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:38.980377 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35784: BlockingRecv error: recv got EOF from 127.0.0.1:35784 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 11:37:38.981252 25318 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:38.979285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51952 (local address 127.2.243.190:39619)
0620 11:37:38.979321 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:38.979323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:38.980756 (+ 1433us) server_negotiation.cc:407] Connection header received
0620 11:37:38.980850 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:38.980851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:38.980861 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:38.980890 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:38.981193 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51952: BlockingRecv error: recv got EOF from 127.0.0.1:51952 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 11:37:38.981982 25469 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:38.979548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52128 (local address 127.2.243.188:37111)
0620 11:37:38.979582 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:38.979584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:38.981488 (+ 1904us) server_negotiation.cc:407] Connection header received
0620 11:37:38.981612 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:38.981613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:38.981623 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:38.981652 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:38.981927 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52128: BlockingRecv error: recv got EOF from 127.0.0.1:52128 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 11:37:38.984277 25394 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:38.983564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35798 (local address 127.2.243.189:33979)
0620 11:37:38.983596 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:38.983598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:38.983803 (+ 205us) server_negotiation.cc:407] Connection header received
0620 11:37:38.983896 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:38.983897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:38.983913 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:38.983940 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:38.984235 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35798: BlockingRecv error: recv got EOF from 127.0.0.1:35798 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 11:37:38.987094 25394 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:38.986366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35802 (local address 127.2.243.189:33979)
0620 11:37:38.986397 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:38.986399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:38.986618 (+ 219us) server_negotiation.cc:407] Connection header received
0620 11:37:38.986705 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:38.986706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:38.986714 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:38.986740 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:38.987057 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35802: BlockingRecv error: recv got EOF from 127.0.0.1:35802 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 11:37:38.990850 25394 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:38.990148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35816 (local address 127.2.243.189:33979)
0620 11:37:38.990180 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:38.990181 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:38.990402 (+ 221us) server_negotiation.cc:407] Connection header received
0620 11:37:38.990491 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:38.990492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:38.990502 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:38.990529 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:38.990811 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35816: BlockingRecv error: recv got EOF from 127.0.0.1:35816 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 11:37:38.996548 25394 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:38.995854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35824 (local address 127.2.243.189:33979)
0620 11:37:38.995886 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:38.995888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:38.996108 (+ 220us) server_negotiation.cc:407] Connection header received
0620 11:37:38.996195 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:38.996195 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:38.996205 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:38.996231 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:38.996506 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35824: BlockingRecv error: recv got EOF from 127.0.0.1:35824 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 11:37:39.006582 25394 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:39.005728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35836 (local address 127.2.243.189:33979)
0620 11:37:39.005769 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:39.005771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:39.006036 (+ 265us) server_negotiation.cc:407] Connection header received
0620 11:37:39.006141 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:39.006142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:39.006152 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:39.006182 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:39.006517 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35836: BlockingRecv error: recv got EOF from 127.0.0.1:35836 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 11:37:39.024865 25394 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:39.023873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35842 (local address 127.2.243.189:33979)
0620 11:37:39.023915 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:39.023917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:39.024220 (+ 303us) server_negotiation.cc:407] Connection header received
0620 11:37:39.024326 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:39.024327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:39.024339 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:39.024369 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:39.024808 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35842: BlockingRecv error: recv got EOF from 127.0.0.1:35842 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 11:37:39.059255 25394 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:39.058238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35844 (local address 127.2.243.189:33979)
0620 11:37:39.058284 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:39.058287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:39.058601 (+ 314us) server_negotiation.cc:407] Connection header received
0620 11:37:39.058712 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:39.058714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:39.058726 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:39.058757 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:39.059197 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35844: BlockingRecv error: recv got EOF from 127.0.0.1:35844 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 11:37:39.125507 25394 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:39.124681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35856 (local address 127.2.243.189:33979)
0620 11:37:39.124747 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:39.124750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:39.124895 (+ 145us) server_negotiation.cc:407] Connection header received
0620 11:37:39.124951 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:39.124952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:39.124981 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:39.125057 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:39.125444 (+ 387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35856: BlockingRecv error: recv got EOF from 127.0.0.1:35856 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 11:37:39.255613 25394 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:39.254663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35860 (local address 127.2.243.189:33979)
0620 11:37:39.254751 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:39.254754 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:39.254858 (+ 104us) server_negotiation.cc:407] Connection header received
0620 11:37:39.254956 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:39.254957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:39.254970 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:39.255157 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:39.255542 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35860: BlockingRecv error: recv got EOF from 127.0.0.1:35860 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 11:37:39.513481 25394 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:39.512803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35874 (local address 127.2.243.189:33979)
0620 11:37:39.512877 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:39.512880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:39.512889 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:37:39.512965 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:39.512966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:39.512977 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:39.513054 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:39.513416 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35874: BlockingRecv error: recv got EOF from 127.0.0.1:35874 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 11:37:39.873872 25482 catalog_manager.cc:1259] Loaded cluster ID: 41c471dc217c456b9a1ba355ef46d755
I20260620 11:37:39.873939 25482 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4d0a1f19824648a2b94f71d3b138a3d1: loading cluster ID for follower catalog manager: success
I20260620 11:37:39.874791 25482 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4d0a1f19824648a2b94f71d3b138a3d1: acquiring CA information for follower catalog manager: success
I20260620 11:37:39.875125 25482 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4d0a1f19824648a2b94f71d3b138a3d1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 11:37:39.957679 25529 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 62dd33fb6ed84132a65011fa70ed92ed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 11:37:40.028975 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:40.026595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35884 (local address 127.2.243.189:33979)
0620 11:37:40.026722 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:40.026724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:40.026734 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:37:40.026791 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:40.026792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:40.026840 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:40.026960 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:40.027095 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:40.027582 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:40.028307 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:40.028499 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:40.028557 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:40.028688 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:40.028796 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:40.028855 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:40.028892 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 11:37:40.028909 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260620 11:37:40.032995 25602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:40.030835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52144 (local address 127.2.243.188:37111)
0620 11:37:40.030974 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:40.030975 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:40.030983 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:37:40.031015 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:40.031016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:40.031055 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:40.031119 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:40.031224 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:40.031708 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:40.032360 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:40.032532 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:40.032578 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:40.032699 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:40.032783 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:40.032833 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:40.032871 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 11:37:40.032886 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260620 11:37:40.037576 25602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:40.034907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52158 (local address 127.2.243.188:37111)
0620 11:37:40.034944 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:40.034946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:40.035076 (+ 130us) server_negotiation.cc:407] Connection header received
0620 11:37:40.035165 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:40.035166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:40.035175 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:40.035204 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:40.035419 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:40.036119 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:40.036848 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:40.037126 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:40.037189 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:40.037331 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:40.037360 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:40.037418 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:40.037465 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 11:37:40.037487 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:37:40.037704 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:40.034633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35896 (local address 127.2.243.189:33979)
0620 11:37:40.034670 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:40.034672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:40.034789 (+ 117us) server_negotiation.cc:407] Connection header received
0620 11:37:40.034913 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:40.034915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:40.034927 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:40.034958 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:40.035193 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:40.035965 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:40.036924 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:40.037212 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:40.037273 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:40.037440 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:40.037471 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:40.037499 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:40.037607 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 11:37:40.037631 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:37:40.037705 25609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:40.034665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51968 (local address 127.2.243.190:39619)
0620 11:37:40.034790 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:40.034792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:40.034837 (+ 45us) server_negotiation.cc:407] Connection header received
0620 11:37:40.034983 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:40.034985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:40.035032 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:40.035105 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:40.035295 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:40.035846 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:40.036943 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:40.037144 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:40.037219 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:40.037412 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:40.037512 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:40.037574 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:40.037599 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 11:37:40.037622 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":53,"threads_started":1}
I20260620 11:37:40.039438 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 25247
I20260620 11:37:40.044836 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 25322
I20260620 11:37:40.050107 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 25397
2026-06-20T11:37:40Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2772 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:37:40 dist-test-slave-ctbw krb5kdc[25613](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:37:40 dist-test-slave-ctbw krb5kdc[25613](info): set up 2 sockets
Jun 20 11:37:40 dist-test-slave-ctbw krb5kdc[25613](info): commencing operation
krb5kdc: starting...
W20260620 11:37:41.099414 3022 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/test-user.keytab.
Jun 20 11:37:41 dist-test-slave-ctbw krb5kdc[25613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:37:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:37:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.2.243.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
Entry for principal kudu/127.2.243.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab.
I20260620 11:37:41.213650 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:46631
--webserver_interface=127.2.243.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:46631,127.2.243.189:41023,127.2.243.188:37245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:37:41.292925 25626 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 11:37:41.293164 25626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:41.293195 25626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:41.293232 25626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:41.294405 25626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:41.294472 25626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:41.294497 25626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:41.294525 25626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:41.294555 25626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:41.296028 25626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:46631,127.2.243.189:41023,127.2.243.188:37245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.190.keytab
--principal=kudu/127.2.243.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.190:46631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.2.243.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:41.296291 25626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:41.296507 25626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:41.301267 25631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:41.301311 25632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:41.301440 25626 server_base.cc:1034] running on GCE node
W20260620 11:37:41.301482 25634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:41.301752 25626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:41.302053 25626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:41.303215 25626 hybrid_clock.cc:648] HybridClock initialized: now 1781955461303195 us; error 32 us; skew 500 ppm
Jun 20 11:37:41 dist-test-slave-ctbw krb5kdc[25613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:41.305578 25626 init.cc:377] Logged in from keytab as kudu/127.2.243.190@KRBTEST.COM (short username kudu)
I20260620 11:37:41.306020 25626 webserver.cc:460] Webserver started at http://127.2.243.190:46667/ using document root <none> and password file <none>
I20260620 11:37:41.306197 25626 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:41.306237 25626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:41.306322 25626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:41.307404 25626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-0/data/instance:
uuid: "5aaf4595a3f74471be8a36719da5b3f9"
format_stamp: "Formatted at 2026-06-20 11:37:41 on dist-test-slave-ctbw"
server_key: "4f18a7fa06bbb70c62469b804934ef67"
server_key_iv: "b0e64c16a64a95c29c832bd0901c5993"
server_key_version: "encryptionkey@0"
I20260620 11:37:41.307765 25626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-0/wal/instance:
uuid: "5aaf4595a3f74471be8a36719da5b3f9"
format_stamp: "Formatted at 2026-06-20 11:37:41 on dist-test-slave-ctbw"
server_key: "4f18a7fa06bbb70c62469b804934ef67"
server_key_iv: "b0e64c16a64a95c29c832bd0901c5993"
server_key_version: "encryptionkey@0"
I20260620 11:37:41.308993 25626 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:37:41.309674 25641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:41.309868 25626 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:37:41.309947 25626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-0/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-0/wal
uuid: "5aaf4595a3f74471be8a36719da5b3f9"
format_stamp: "Formatted at 2026-06-20 11:37:41 on dist-test-slave-ctbw"
server_key: "4f18a7fa06bbb70c62469b804934ef67"
server_key_iv: "b0e64c16a64a95c29c832bd0901c5993"
server_key_version: "encryptionkey@0"
I20260620 11:37:41.309998 25626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:41.316015 25626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:41.317669 25626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:41.317804 25626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:41.321441 25626 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.190:46631
I20260620 11:37:41.321512 25693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.190:46631 every 8 connection(s)
I20260620 11:37:41.321839 25626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-0/data/info.pb
I20260620 11:37:41.322801 25694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:41.324266 25694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46631 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:37:41.328039 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 25626
I20260620 11:37:41.328191 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-0/wal/instance
I20260620 11:37:41.328408 3022 external_mini_cluster.cc:1428] Setting key 65328dd02c919d26486cb1aa631ec54d
Jun 20 11:37:41 dist-test-slave-ctbw krb5kdc[25613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:37:41.333729 25697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.325057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33418 (local address 127.2.243.190:46631)
0620 11:37:41.325182 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.325186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.325213 (+ 27us) server_negotiation.cc:407] Connection header received
0620 11:37:41.325676 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.325678 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.325694 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.325769 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.326272 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.327132 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.327942 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.328435 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.328546 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.330639 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:41.330644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:41.330645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:41.330650 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:41.332196 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.332657 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.332659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.332659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.332731 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.332980 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.332980 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.332980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.333172 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:41.333232 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.333467 (+ 235us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.333580 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":90,"thread_start_us":64,"threads_started":1}
I20260620 11:37:41.333830 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.324912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:46631 (local address 127.0.0.1:33418)
0620 11:37:41.325053 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.325087 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:41.325534 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:41.325797 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:41.325799 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:41.326022 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:41.326186 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:41.326190 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.327211 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.327212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:41.327864 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:41.327866 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.328551 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.328557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:41.328641 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.329174 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:41.329182 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:41.330554 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:41.332279 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:41.332285 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:41.332287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:41.332566 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:41.332797 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:41.332798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:41.332798 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:41.332910 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:41.333244 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:41.333251 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:41.333328 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 11:37:41.333446 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 11:37:41.333678 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":64,"thread_start_us":59,"threads_started":1}
I20260620 11:37:41.334494 25694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:37:41.334928 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.334753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:41023 (local address 127.0.0.1:60410)
0620 11:37:41.334807 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.334855 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.189:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 11:37:41.335080 25642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.243.189:41023: connect: Connection refused (error 111)
I20260620 11:37:41.335489 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.335382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:41023 (local address 127.0.0.1:60412)
0620 11:37:41.335432 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.335461 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.189:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 11:37:41.335599 25694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.189:41023: Network error: Client connection negotiation failed: client connection to 127.2.243.189:41023: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.2.243.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.189.keytab.
Entry for principal kudu/127.2.243.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.189.keytab.
I20260620 11:37:41.353698 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.189:41023
--webserver_interface=127.2.243.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.189.keytab
--principal=kudu/127.2.243.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:46631,127.2.243.189:41023,127.2.243.188:37245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:37:41.384833 25694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } attempt: 1
I20260620 11:37:41.385385 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.385239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:41023 (local address 127.0.0.1:60426)
0620 11:37:41.385300 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.385348 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.189:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 11:37:41.385835 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.385746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:41023 (local address 127.0.0.1:60428)
0620 11:37:41.385791 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.385817 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.189:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 11:37:41.385938 25694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.189:41023: Network error: Client connection negotiation failed: client connection to 127.2.243.189:41023: connect: Connection refused (error 111)
W20260620 11:37:41.433815 25701 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 11:37:41.434016 25701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:41.434033 25701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:41.434053 25701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:41.435113 25701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:41.435153 25701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:41.435168 25701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:41.435180 25701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:41.435192 25701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:41.436543 25701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:46631,127.2.243.189:41023,127.2.243.188:37245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.189.keytab
--principal=kudu/127.2.243.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.189:41023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.2.243.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:41.436720 25701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:41.436882 25701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:37:41.441733 25709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:41.441777 25707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:41.441871 25706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:41.441777 25701 server_base.cc:1034] running on GCE node
I20260620 11:37:41.442173 25701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:41.442464 25701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:41.443631 25701 hybrid_clock.cc:648] HybridClock initialized: now 1781955461443624 us; error 37 us; skew 500 ppm
Jun 20 11:37:41 dist-test-slave-ctbw krb5kdc[25613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:41.445992 25701 init.cc:377] Logged in from keytab as kudu/127.2.243.189@KRBTEST.COM (short username kudu)
I20260620 11:37:41.446447 25701 webserver.cc:460] Webserver started at http://127.2.243.189:34703/ using document root <none> and password file <none>
I20260620 11:37:41.446671 25701 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:41.446740 25701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:41.446933 25701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:41.447983 25701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-1/data/instance:
uuid: "092f213c9e554dfab27930382e7402a5"
format_stamp: "Formatted at 2026-06-20 11:37:41 on dist-test-slave-ctbw"
server_key: "6255e01aae6a084590c556cb11b85d76"
server_key_iv: "334336f7d113fbe487823f00591d52e3"
server_key_version: "encryptionkey@0"
I20260620 11:37:41.448344 25701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-1/wal/instance:
uuid: "092f213c9e554dfab27930382e7402a5"
format_stamp: "Formatted at 2026-06-20 11:37:41 on dist-test-slave-ctbw"
server_key: "6255e01aae6a084590c556cb11b85d76"
server_key_iv: "334336f7d113fbe487823f00591d52e3"
server_key_version: "encryptionkey@0"
I20260620 11:37:41.449674 25701 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:37:41.450388 25716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:41.450565 25701 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:37:41.450696 25701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-1/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-1/wal
uuid: "092f213c9e554dfab27930382e7402a5"
format_stamp: "Formatted at 2026-06-20 11:37:41 on dist-test-slave-ctbw"
server_key: "6255e01aae6a084590c556cb11b85d76"
server_key_iv: "334336f7d113fbe487823f00591d52e3"
server_key_version: "encryptionkey@0"
I20260620 11:37:41.450788 25701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:41.454068 25694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } attempt: 2
I20260620 11:37:41.454479 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.454363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:41023 (local address 127.0.0.1:60440)
0620 11:37:41.454418 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.454454 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.189:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 11:37:41.454846 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.454749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:41023 (local address 127.0.0.1:60442)
0620 11:37:41.454796 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.454827 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.189:41023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 11:37:41.454957 25694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.189:41023: Network error: Client connection negotiation failed: client connection to 127.2.243.189:41023: connect: Connection refused (error 111)
I20260620 11:37:41.459622 25701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:41.461494 25701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:41.461671 25701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:41.466013 25701 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.189:41023
I20260620 11:37:41.466039 25768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.189:41023 every 8 connection(s)
I20260620 11:37:41.466538 25701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-1/data/info.pb
I20260620 11:37:41.467710 25769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:41.468459 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 25701
I20260620 11:37:41.468580 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-1/wal/instance
I20260620 11:37:41.468840 3022 external_mini_cluster.cc:1428] Setting key 487fca308440226fbaef7ce13b92775c
I20260620 11:37:41.470207 25769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46631 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.2.243.188@KRBTEST.COM; defaulting to no policy
Jun 20 11:37:41 dist-test-slave-ctbw krb5kdc[25613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.189@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.2.243.188@KRBTEST.COM" created.
I20260620 11:37:41.480707 25697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.471181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33420 (local address 127.2.243.190:46631)
0620 11:37:41.471255 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.471258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.471516 (+ 258us) server_negotiation.cc:407] Connection header received
0620 11:37:41.472218 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.472219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.472232 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.472291 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.473163 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.473649 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.475273 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.475468 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.475526 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.477956 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:41.477958 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:41.477958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:41.477966 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:41.479277 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.479706 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.479707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.479707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.479794 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.480050 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.480051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.480051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.480175 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:41.480236 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.480566 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.480642 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 11:37:41.481024 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.471132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:46631 (local address 127.0.0.1:33420)
0620 11:37:41.471348 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.471397 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:41.472106 (+ 709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:41.472357 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:41.472361 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:41.472757 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:41.473058 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:41.473064 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.473748 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.473750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:41.475185 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:41.475188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.475565 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.475567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:41.475730 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.476277 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:41.476288 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:41.477855 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:41.479382 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:41.479390 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:41.479392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:41.479609 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:41.479848 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:41.479849 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:41.479849 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:41.479955 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:41.480247 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:41.480253 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:41.480425 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 11:37:41.480549 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 11:37:41.480742 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":102,"threads_started":1}
I20260620 11:37:41.481585 25769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 11:37:41 dist-test-slave-ctbw krb5kdc[25613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.189@KRBTEST.COM for kudu/127.2.243.189@KRBTEST.COM
I20260620 11:37:41.490173 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.481918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60448 (local address 127.2.243.189:41023)
0620 11:37:41.482079 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.482083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.482096 (+ 13us) server_negotiation.cc:407] Connection header received
0620 11:37:41.482114 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.482116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.482126 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.482192 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.482527 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.483262 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.484364 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.484981 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.485071 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.487158 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:41.487166 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:41.487167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:41.487174 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:41.488913 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.489302 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.489306 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.489306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.489385 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.489585 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.489586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.489586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.489733 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:41.489802 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.489916 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.490057 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
I20260620 11:37:41.490145 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.481858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:41023 (local address 127.0.0.1:60448)
0620 11:37:41.481911 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.481922 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:41.481962 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:41.482219 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:41.482220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:41.482380 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:41.482440 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:41.482442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.483346 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.483348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:41.484266 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:41.484269 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.485089 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.485090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:41.485187 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.485630 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:41.485636 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:41.487042 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:41.489022 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:41.489023 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:41.489023 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:41.489205 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:41.489461 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:41.489462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:41.489462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:41.489520 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:41.489793 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:41.489794 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:41.489827 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 11:37:41.489877 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 11:37:41.489993 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 11:37:41.490846 25769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:37:41.491232 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.491119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37245 (local address 127.0.0.1:47508)
0620 11:37:41.491173 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.491215 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 11:37:41.491380 25717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37245: connect: Connection refused (error 111)
I20260620 11:37:41.491822 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.491708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37245 (local address 127.0.0.1:47522)
0620 11:37:41.491768 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.491796 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 11:37:41.491936 25769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.188:37245: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37245: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.2.243.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.188.keytab.
Entry for principal kudu/127.2.243.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.188.keytab.
I20260620 11:37:41.495679 3022 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDepOyZ/build/release/bin/kudu
/tmp/dist-test-taskDepOyZ/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.188:37245
--webserver_interface=127.2.243.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.188.keytab
--principal=kudu/127.2.243.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.2.243.148:45223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.2.243.190:46631,127.2.243.189:41023,127.2.243.188:37245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:37:41.541175 25769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } attempt: 1
I20260620 11:37:41.541707 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.541560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37245 (local address 127.0.0.1:47534)
0620 11:37:41.541634 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.541679 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 11:37:41.542114 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.542020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37245 (local address 127.0.0.1:47542)
0620 11:37:41.542062 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.542097 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 11:37:41.542258 25769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.188:37245: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37245: connect: Connection refused (error 111)
I20260620 11:37:41.546135 25694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } attempt: 3
Jun 20 11:37:41 dist-test-slave-ctbw krb5kdc[25613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for kudu/127.2.243.189@KRBTEST.COM
I20260620 11:37:41.553618 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.546457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:41023 (local address 127.0.0.1:60458)
0620 11:37:41.546518 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.546530 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:41.546585 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:41.546785 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:41.546785 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:41.546988 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:41.547050 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:41.547052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.548008 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.548009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:41.548541 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:41.548543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.549066 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.549067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:41.549155 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.549659 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:41.549665 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:41.550830 (+ 1165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:41.552592 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:41.552593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:41.552594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:41.552802 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:41.553011 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:41.553011 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:41.553012 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:41.553062 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:41.553353 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:41.553354 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:41.553401 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 11:37:41.553474 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 11:37:41.553549 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 11:37:41.553776 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.546579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60458 (local address 127.2.243.189:41023)
0620 11:37:41.546639 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.546641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.546651 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:37:41.546678 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.546679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.546692 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.546764 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.547124 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.547905 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.548624 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.548956 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.549051 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.550966 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:41.550968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:41.550968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:41.550976 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:41.552504 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.552871 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.552872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.552873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.552935 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.553126 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.553127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.553127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.553260 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:41.553328 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.553518 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.553621 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 11:37:41.554359 25694 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:37:41.554705 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.554599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37245 (local address 127.0.0.1:47544)
0620 11:37:41.554649 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.554675 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 11:37:41.555096 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.555015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37245 (local address 127.0.0.1:47560)
0620 11:37:41.555059 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.555082 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 11:37:41.555207 25694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.188:37245: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37245: connect: Connection refused (error 111)
W20260620 11:37:41.579090 25776 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 11:37:41.579291 25776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:37:41.579308 25776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:37:41.579355 25776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:37:41.580485 25776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:37:41.580524 25776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:37:41.580539 25776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:37:41.580585 25776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:37:41.580613 25776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:37:41.582043 25776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.2.243.148:45223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.2.243.190:46631,127.2.243.189:41023,127.2.243.188:37245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/krb5kdc/kudu_127.2.243.188.keytab
--principal=kudu/127.2.243.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.2.243.188:37245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.2.243.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:37:41.582288 25776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:37:41.582509 25776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 11:37:41.586694 25694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } attempt: 1
I20260620 11:37:41.587138 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.586982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37245 (local address 127.0.0.1:47562)
0620 11:37:41.587044 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.587092 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 11:37:41.587549 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.587456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37245 (local address 127.0.0.1:47566)
0620 11:37:41.587490 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.587518 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260620 11:37:41.587582 25782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:41.587586 25781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:37:41.587759 25694 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.188:37245: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37245: connect: Connection refused (error 111)
W20260620 11:37:41.587779 25784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:37:41.588071 25776 server_base.cc:1034] running on GCE node
I20260620 11:37:41.588358 25776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:37:41.588630 25776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:37:41.589783 25776 hybrid_clock.cc:648] HybridClock initialized: now 1781955461589771 us; error 30 us; skew 500 ppm
Jun 20 11:37:41 dist-test-slave-ctbw krb5kdc[25613](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:37:41.592243 25776 init.cc:377] Logged in from keytab as kudu/127.2.243.188@KRBTEST.COM (short username kudu)
I20260620 11:37:41.592599 25776 webserver.cc:460] Webserver started at http://127.2.243.188:37039/ using document root <none> and password file <none>
I20260620 11:37:41.592772 25776 fs_manager.cc:359] Metadata directory not provided
I20260620 11:37:41.592803 25776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:37:41.592893 25776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:37:41.593848 25776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-2/data/instance:
uuid: "2ca3db165b7e4264a1f3849cf839bca0"
format_stamp: "Formatted at 2026-06-20 11:37:41 on dist-test-slave-ctbw"
server_key: "9db36e9f2e097106898409578884f9b5"
server_key_iv: "ecf5ce1f8f429c9db73c41b163678bcf"
server_key_version: "encryptionkey@0"
I20260620 11:37:41.594240 25776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-2/wal/instance:
uuid: "2ca3db165b7e4264a1f3849cf839bca0"
format_stamp: "Formatted at 2026-06-20 11:37:41 on dist-test-slave-ctbw"
server_key: "9db36e9f2e097106898409578884f9b5"
server_key_iv: "ecf5ce1f8f429c9db73c41b163678bcf"
server_key_version: "encryptionkey@0"
I20260620 11:37:41.595491 25776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 11:37:41.596179 25791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:37:41.596372 25776 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:37:41.596441 25776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-2/data,/tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-2/wal
uuid: "2ca3db165b7e4264a1f3849cf839bca0"
format_stamp: "Formatted at 2026-06-20 11:37:41 on dist-test-slave-ctbw"
server_key: "9db36e9f2e097106898409578884f9b5"
server_key_iv: "ecf5ce1f8f429c9db73c41b163678bcf"
server_key_version: "encryptionkey@0"
I20260620 11:37:41.596524 25776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:37:41.606455 25776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:37:41.608223 25776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:37:41.608364 25776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:37:41.610409 25769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } attempt: 2
I20260620 11:37:41.610839 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.610687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37245 (local address 127.0.0.1:47580)
0620 11:37:41.610755 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.610797 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 11:37:41.611266 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.611135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37245 (local address 127.0.0.1:47590)
0620 11:37:41.611204 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.611232 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 11:37:41.611371 25769 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.2.243.188:37245: Network error: Client connection negotiation failed: client connection to 127.2.243.188:37245: connect: Connection refused (error 111)
I20260620 11:37:41.612635 25776 rpc_server.cc:307] RPC server started. Bound to: 127.2.243.188:37245
I20260620 11:37:41.612682 25843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.2.243.188:37245 every 8 connection(s)
I20260620 11:37:41.613027 25776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-2/data/info.pb
I20260620 11:37:41.613962 25844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:37:41.615523 25844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46631 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:37:41.620162 3022 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDepOyZ/build/release/bin/kudu as pid 25776
I20260620 11:37:41.620286 3022 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDepOyZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955192827595-3022-0/minicluster-data/master-2/wal/instance
I20260620 11:37:41.620510 3022 external_mini_cluster.cc:1428] Setting key b79944b504235b2ca3ae237da2aed39f
Jun 20 11:37:41 dist-test-slave-ctbw krb5kdc[25613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.188@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
I20260620 11:37:41.624670 25697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.616208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33422 (local address 127.2.243.190:46631)
0620 11:37:41.616252 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.616256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.616449 (+ 193us) server_negotiation.cc:407] Connection header received
0620 11:37:41.616964 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.616966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.616979 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.617029 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.617626 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.618089 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.619167 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.619357 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.619463 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.621788 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:41.621802 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:41.621802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:41.621810 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:41.623365 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.623739 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.623740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.623741 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.623803 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.624039 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.624040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.624040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.624165 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:41.624239 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.624501 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.624584 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:37:41.624854 25846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.616137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.190:46631 (local address 127.0.0.1:33422)
0620 11:37:41.616334 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.616368 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:41.616848 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:41.617093 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:41.617097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:41.617342 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:41.617522 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:41.617526 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.618198 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.618199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:41.619075 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:41.619077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.619484 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.619485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:41.619611 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.620079 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:41.620090 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:41.621699 (+ 1609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:41.623445 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:41.623450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:41.623451 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:41.623672 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:41.623868 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:41.623869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:41.623869 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:41.623960 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:41.624243 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:41.624249 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:41.624366 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 11:37:41.624466 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 11:37:41.624601 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":98,"threads_started":1}
I20260620 11:37:41.625468 25844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 11:37:41 dist-test-slave-ctbw krb5kdc[25613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.188@KRBTEST.COM
Jun 20 11:37:41 dist-test-slave-ctbw krb5kdc[25613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.188@KRBTEST.COM for kudu/127.2.243.189@KRBTEST.COM
I20260620 11:37:41.629845 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.620901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47594 (local address 127.2.243.188:37245)
0620 11:37:41.621042 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.621046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.621208 (+ 162us) server_negotiation.cc:407] Connection header received
0620 11:37:41.621274 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.621276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.621297 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.621386 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.622068 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.623001 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.623923 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.624234 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.624324 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.626846 (+ 2522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:41.626852 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:41.626852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:41.626859 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:41.628655 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.629017 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.629019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.629020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.629076 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.629285 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.629286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.629286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.629442 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:41.629593 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.629704 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.629758 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":93,"threads_started":1}
I20260620 11:37:41.632524 25846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.625690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.189:41023 (local address 127.0.0.1:60464)
0620 11:37:41.625733 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.625743 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:41.625802 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:41.625930 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:41.625931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:41.626059 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:41.626104 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:41.626105 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.626800 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.626801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:41.627338 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:41.627340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.627874 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.627875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:41.627950 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.628330 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:41.628334 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:41.629537 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:41.631527 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:41.631528 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:41.631529 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:41.631698 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:41.631905 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:41.631906 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:41.631907 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:41.631961 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:41.632221 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:41.632221 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:41.632261 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 11:37:41.632349 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 11:37:41.632434 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 11:37:41.632529 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.625760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60464 (local address 127.2.243.189:41023)
0620 11:37:41.625825 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.625827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.625836 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:37:41.625860 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.625861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.625871 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.625926 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.626206 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.626711 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.627432 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.627748 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.627873 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.629616 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:41.629617 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:41.629617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:41.629624 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:41.631469 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.631773 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.631774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.631774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.631829 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.632033 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.632033 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.632034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.632152 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:41.632228 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.632377 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.632448 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 11:37:41.632901 25844 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 11:37:41 dist-test-slave-ctbw krb5kdc[25613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.190@KRBTEST.COM
Jun 20 11:37:41 dist-test-slave-ctbw krb5kdc[25613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.188@KRBTEST.COM for kudu/127.2.243.188@KRBTEST.COM
I20260620 11:37:41.637703 25697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.630766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.2.243.190:46631)
0620 11:37:41.630800 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.630802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.630809 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:37:41.630830 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.630831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.630839 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.630940 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.631482 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.631907 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.632767 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.633042 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.633137 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.635192 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:41.635193 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:41.635193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:41.635199 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:41.636600 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.636966 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.636967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.636967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.637022 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.637223 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.637224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.637224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.637316 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:41.637479 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.637518 (+ 39us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.637586 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 11:37:41.639694 25846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.633134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37245 (local address 127.0.0.1:47600)
0620 11:37:41.633202 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.633211 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:41.633253 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:41.633590 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:41.633590 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:41.633718 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:41.633740 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:41.633741 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.634388 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.634389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:41.634935 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:41.634937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.635384 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.635386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:41.635438 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.635734 (+ 296us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:41.635739 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:41.636914 (+ 1175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:41.638696 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:41.638697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:41.638697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:41.638953 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:41.639200 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:41.639200 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:41.639200 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:41.639255 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:41.639495 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:41.639495 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:41.639524 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 11:37:41.639570 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 11:37:41.639635 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260620 11:37:41.639881 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.633160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47600 (local address 127.2.243.188:37245)
0620 11:37:41.633190 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.633192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.633240 (+ 48us) server_negotiation.cc:407] Connection header received
0620 11:37:41.633507 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.633508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.633517 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.633588 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.633822 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.634315 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.635007 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.635304 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.635353 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.637016 (+ 1663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:41.637017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:41.637017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:41.637023 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:41.638620 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.639055 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.639056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.639057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.639114 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.639315 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.639316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.639316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.639434 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:41.639497 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.639590 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.639692 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":15}
I20260620 11:37:41.641462 25844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca3db165b7e4264a1f3849cf839bca0: Bootstrap starting.
I20260620 11:37:41.642360 25844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ca3db165b7e4264a1f3849cf839bca0: Neither blocks nor log segments found. Creating new log.
Jun 20 11:37:41 dist-test-slave-ctbw krb5kdc[25613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.2.243.189@KRBTEST.COM
I20260620 11:37:41.642679 25844 log.cc:824] T 00000000000000000000000000000000 P 2ca3db165b7e4264a1f3849cf839bca0: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:41.643464 25844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ca3db165b7e4264a1f3849cf839bca0: No bootstrap required, opened a new log
I20260620 11:37:41.645398 25844 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ca3db165b7e4264a1f3849cf839bca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aaf4595a3f74471be8a36719da5b3f9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46631 } } peers { permanent_uuid: "092f213c9e554dfab27930382e7402a5" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } } peers { permanent_uuid: "2ca3db165b7e4264a1f3849cf839bca0" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } }
I20260620 11:37:41.645561 25844 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ca3db165b7e4264a1f3849cf839bca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ca3db165b7e4264a1f3849cf839bca0, State: Initialized, Role: FOLLOWER
I20260620 11:37:41.645622 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.638353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60466 (local address 127.2.243.189:41023)
0620 11:37:41.638392 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.638393 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.638402 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:37:41.638434 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.638434 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.638443 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.638486 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.639078 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.639800 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.640603 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.640835 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.640907 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.643225 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:41.643226 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:41.643226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:41.643232 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:41.644547 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.644938 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.644939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.644939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.644983 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.645166 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.645167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.645167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.645263 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:41.645360 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.645466 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.645519 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:37:41.645790 25844 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ca3db165b7e4264a1f3849cf839bca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aaf4595a3f74471be8a36719da5b3f9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46631 } } peers { permanent_uuid: "092f213c9e554dfab27930382e7402a5" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } } peers { permanent_uuid: "2ca3db165b7e4264a1f3849cf839bca0" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } }
I20260620 11:37:41.646106 25850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca3db165b7e4264a1f3849cf839bca0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aaf4595a3f74471be8a36719da5b3f9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46631 } } peers { permanent_uuid: "092f213c9e554dfab27930382e7402a5" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } } peers { permanent_uuid: "2ca3db165b7e4264a1f3849cf839bca0" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } } }
I20260620 11:37:41.646183 25844 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ca3db165b7e4264a1f3849cf839bca0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:41.646204 25850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca3db165b7e4264a1f3849cf839bca0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:37:41.647097 3022 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 11:37:41.647990 25861 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ca3db165b7e4264a1f3849cf839bca0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:37:41.648043 25861 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:37:41.648245 25668 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5aaf4595a3f74471be8a36719da5b3f9"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33428
I20260620 11:37:41.648306 25776 master_runner.cc:186] Error getting master registration for 127.2.243.190:46631: OK, instance_id { permanent_uuid: "5aaf4595a3f74471be8a36719da5b3f9" instance_seqno: 1781955461320486 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 11:37:41.651034 25776 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 11:37:41.662904 25694 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } attempt: 2
Jun 20 11:37:41 dist-test-slave-ctbw krb5kdc[25613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.190@KRBTEST.COM for kudu/127.2.243.188@KRBTEST.COM
I20260620 11:37:41.670696 25696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.663178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37245 (local address 127.0.0.1:47604)
0620 11:37:41.663234 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.663247 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:41.663303 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:41.663430 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:41.663431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:41.663610 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:41.663678 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:41.663680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.664561 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.664562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:41.665372 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:41.665374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.665809 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.665810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:41.665870 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.666347 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:41.666353 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:41.667798 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:41.669597 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:41.669598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:41.669599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:41.669833 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:41.670025 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:41.670026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:41.670026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:41.670092 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:41.670396 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:41.670397 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:41.670441 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 11:37:41.670511 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 11:37:41.670598 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 11:37:41.670714 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.663240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47604 (local address 127.2.243.188:37245)
0620 11:37:41.663301 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.663303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.663312 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:37:41.663347 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.663348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.663356 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.663398 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.663770 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.664481 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.665453 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.665735 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.665795 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.667901 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:41.667902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:41.667902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:41.667909 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:41.669515 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.669904 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.669904 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.669905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.669962 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.670159 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.670159 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.670160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.670324 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:41.670391 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.670538 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.670610 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 11:37:41.672284 25694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9: Bootstrap starting.
I20260620 11:37:41.673013 25694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:41.673292 25694 log.cc:824] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:41.674026 25694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9: No bootstrap required, opened a new log
I20260620 11:37:41.675104 25694 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aaf4595a3f74471be8a36719da5b3f9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46631 } } peers { permanent_uuid: "092f213c9e554dfab27930382e7402a5" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } } peers { permanent_uuid: "2ca3db165b7e4264a1f3849cf839bca0" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } }
I20260620 11:37:41.675232 25694 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aaf4595a3f74471be8a36719da5b3f9, State: Initialized, Role: FOLLOWER
I20260620 11:37:41.675324 25694 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aaf4595a3f74471be8a36719da5b3f9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46631 } } peers { permanent_uuid: "092f213c9e554dfab27930382e7402a5" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } } peers { permanent_uuid: "2ca3db165b7e4264a1f3849cf839bca0" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } }
I20260620 11:37:41.675611 25864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aaf4595a3f74471be8a36719da5b3f9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46631 } } peers { permanent_uuid: "092f213c9e554dfab27930382e7402a5" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } } peers { permanent_uuid: "2ca3db165b7e4264a1f3849cf839bca0" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } } }
I20260620 11:37:41.675678 25694 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:41.675740 25864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:37:41.675752 25668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 11:37:41.675910 25668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:41.676649 25668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aaf4595a3f74471be8a36719da5b3f9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46631 } } peers { permanent_uuid: "092f213c9e554dfab27930382e7402a5" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } } peers { permanent_uuid: "2ca3db165b7e4264a1f3849cf839bca0" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } }
I20260620 11:37:41.676873 25668 leader_election.cc:288] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9 [CANDIDATE]: Term 1 election: Requested vote from peers 092f213c9e554dfab27930382e7402a5 (127.2.243.189:41023), 2ca3db165b7e4264a1f3849cf839bca0 (127.2.243.188:37245)
I20260620 11:37:41.678056 25818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5aaf4595a3f74471be8a36719da5b3f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2ca3db165b7e4264a1f3849cf839bca0"
I20260620 11:37:41.678170 25818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ca3db165b7e4264a1f3849cf839bca0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:41.679045 25818 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2ca3db165b7e4264a1f3849cf839bca0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aaf4595a3f74471be8a36719da5b3f9 in term 1.
I20260620 11:37:41.679267 25642 leader_election.cc:302] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca3db165b7e4264a1f3849cf839bca0, 5aaf4595a3f74471be8a36719da5b3f9; no voters:
I20260620 11:37:41.679381 25864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:37:41.679518 25864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9 [term 1 LEADER]: Becoming Leader. State: Replica: 5aaf4595a3f74471be8a36719da5b3f9, State: Running, Role: LEADER
I20260620 11:37:41.679600 25864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aaf4595a3f74471be8a36719da5b3f9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46631 } } peers { permanent_uuid: "092f213c9e554dfab27930382e7402a5" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } } peers { permanent_uuid: "2ca3db165b7e4264a1f3849cf839bca0" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } }
I20260620 11:37:41.680047 25880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aaf4595a3f74471be8a36719da5b3f9. Latest consensus state: current_term: 1 leader_uuid: "5aaf4595a3f74471be8a36719da5b3f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aaf4595a3f74471be8a36719da5b3f9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46631 } } peers { permanent_uuid: "092f213c9e554dfab27930382e7402a5" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } } peers { permanent_uuid: "2ca3db165b7e4264a1f3849cf839bca0" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } } }
I20260620 11:37:41.680269 25880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:41.678290 25626 master_runner.cc:186] Error getting master registration for 127.2.243.189:41023: OK, instance_id { permanent_uuid: "092f213c9e554dfab27930382e7402a5" instance_seqno: 1781955461464771 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 11:37:41.681167 25626 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 11:37:41.681308 25875 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:37:41.681357 25875 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:37:41.681439 25883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:37:41.681763 25883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:37:41.683483 25818 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2ca3db165b7e4264a1f3849cf839bca0 [term 1 FOLLOWER]: Refusing update from remote peer 5aaf4595a3f74471be8a36719da5b3f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:37:41.683795 25864 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca3db165b7e4264a1f3849cf839bca0" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:37:41.685053 25850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca3db165b7e4264a1f3849cf839bca0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aaf4595a3f74471be8a36719da5b3f9. Latest consensus state: current_term: 1 leader_uuid: "5aaf4595a3f74471be8a36719da5b3f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aaf4595a3f74471be8a36719da5b3f9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46631 } } peers { permanent_uuid: "092f213c9e554dfab27930382e7402a5" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } } peers { permanent_uuid: "2ca3db165b7e4264a1f3849cf839bca0" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } } }
I20260620 11:37:41.685155 25850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca3db165b7e4264a1f3849cf839bca0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:37:41.687449 25864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5aaf4595a3f74471be8a36719da5b3f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aaf4595a3f74471be8a36719da5b3f9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46631 } } peers { permanent_uuid: "092f213c9e554dfab27930382e7402a5" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } } peers { permanent_uuid: "2ca3db165b7e4264a1f3849cf839bca0" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } } }
I20260620 11:37:41.687528 25864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:41.687973 25850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ca3db165b7e4264a1f3849cf839bca0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5aaf4595a3f74471be8a36719da5b3f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aaf4595a3f74471be8a36719da5b3f9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46631 } } peers { permanent_uuid: "092f213c9e554dfab27930382e7402a5" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } } peers { permanent_uuid: "2ca3db165b7e4264a1f3849cf839bca0" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } } }
I20260620 11:37:41.688139 25850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ca3db165b7e4264a1f3849cf839bca0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:37:41.688838 25883 catalog_manager.cc:1347] Generated new cluster ID: 95696643d71c4412b56a8f368fc8c490
I20260620 11:37:41.688882 25883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:37:41.690624 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.680194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60478 (local address 127.2.243.189:41023)
0620 11:37:41.680237 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.680239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.680995 (+ 756us) server_negotiation.cc:407] Connection header received
0620 11:37:41.681146 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.681148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.681160 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.681205 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.682145 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.682918 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.684049 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.684376 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.684437 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.685955 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:41.685957 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:41.685957 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:41.685965 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:41.688328 (+ 2363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.688755 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.688756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.688756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.688820 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.689168 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.689169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.689170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.689327 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:41.689926 (+ 599us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.690444 (+ 518us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.690532 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:37:41.696004 25883 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:37:41.696776 25883 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:37:41.698585 25697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.680696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33440 (local address 127.2.243.190:46631)
0620 11:37:41.680745 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.680747 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.688042 (+ 7295us) server_negotiation.cc:407] Connection header received
0620 11:37:41.688747 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.688748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.688759 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.688790 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.691186 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.691882 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.692821 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.693112 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.693178 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.695204 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:41.695205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:41.695206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:41.695212 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:41.697328 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.697846 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.697847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.697847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.697897 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.698063 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.698064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.698064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.698162 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:41.698231 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.698401 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.698483 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 11:37:41.699213 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.680499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47606 (local address 127.2.243.188:37245)
0620 11:37:41.680544 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.680546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.687793 (+ 7247us) server_negotiation.cc:407] Connection header received
0620 11:37:41.687886 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.687887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.687898 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.687926 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.689474 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.690207 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.692850 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.693144 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.693184 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.695417 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:41.695418 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:41.695418 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:41.695436 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:41.697070 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.697938 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.697939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.697940 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.698002 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.698348 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.698349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.698350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.698472 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:41.698544 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.699042 (+ 498us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.699117 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 11:37:41.701146 25883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9: Generated new TSK 0
I20260620 11:37:41.701313 25883 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:37:41.702507 25769 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } attempt: 3
Jun 20 11:37:41 dist-test-slave-ctbw krb5kdc[25613](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955461, etypes {rep=17 tkt=17 ses=17}, kudu/127.2.243.189@KRBTEST.COM for kudu/127.2.243.188@KRBTEST.COM
I20260620 11:37:41.713220 25847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.702788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47612 (local address 127.2.243.188:37245)
0620 11:37:41.702823 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.702825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.702851 (+ 26us) server_negotiation.cc:407] Connection header received
0620 11:37:41.702952 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.702953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.702962 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.703018 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.703322 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.704008 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.705063 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.705380 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.705450 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.707558 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:37:41.707559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:37:41.707560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:37:41.707566 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:37:41.709110 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.712162 (+ 3052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.712163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.712163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.712235 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:37:41.712513 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:37:41.712514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:37:41.712514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:37:41.712654 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:37:41.712712 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.713031 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.713097 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 11:37:41.713620 25772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.702748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.2.243.188:37245 (local address 127.0.0.1:47612)
0620 11:37:41.702800 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 11:37:41.702812 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:37:41.702885 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:37:41.703024 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:37:41.703025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:37:41.703213 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:37:41.703258 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:41.703260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.704085 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.704086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:41.704988 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:37:41.704991 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.705465 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.705466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:37:41.705536 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.705966 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:37:41.705971 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:37:41.707462 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:37:41.709188 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:41.709189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:41.709189 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:41.709403 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:41.712347 (+ 2944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:37:41.712349 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:37:41.712349 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:37:41.712431 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:37:41.712772 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:37:41.712773 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:37:41.712831 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 11:37:41.712892 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 11:37:41.713507 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 11:37:41.715330 25769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 092f213c9e554dfab27930382e7402a5: Bootstrap starting.
I20260620 11:37:41.716044 25769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 092f213c9e554dfab27930382e7402a5: Neither blocks nor log segments found. Creating new log.
I20260620 11:37:41.716325 25769 log.cc:824] T 00000000000000000000000000000000 P 092f213c9e554dfab27930382e7402a5: Log is configured to *not* fsync() on all Append() calls
I20260620 11:37:41.717229 25769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 092f213c9e554dfab27930382e7402a5: No bootstrap required, opened a new log
I20260620 11:37:41.719286 25769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 092f213c9e554dfab27930382e7402a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aaf4595a3f74471be8a36719da5b3f9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46631 } } peers { permanent_uuid: "092f213c9e554dfab27930382e7402a5" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } } peers { permanent_uuid: "2ca3db165b7e4264a1f3849cf839bca0" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } }
I20260620 11:37:41.719413 25769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 092f213c9e554dfab27930382e7402a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 092f213c9e554dfab27930382e7402a5, State: Initialized, Role: FOLLOWER
I20260620 11:37:41.719485 25769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 092f213c9e554dfab27930382e7402a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aaf4595a3f74471be8a36719da5b3f9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46631 } } peers { permanent_uuid: "092f213c9e554dfab27930382e7402a5" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } } peers { permanent_uuid: "2ca3db165b7e4264a1f3849cf839bca0" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } }
I20260620 11:37:41.719769 25895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 092f213c9e554dfab27930382e7402a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aaf4595a3f74471be8a36719da5b3f9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46631 } } peers { permanent_uuid: "092f213c9e554dfab27930382e7402a5" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } } peers { permanent_uuid: "2ca3db165b7e4264a1f3849cf839bca0" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } } }
I20260620 11:37:41.720017 25895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 092f213c9e554dfab27930382e7402a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:37:41.719854 25742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 092f213c9e554dfab27930382e7402a5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:37:41.720876 25742 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 092f213c9e554dfab27930382e7402a5 [term 1 FOLLOWER]: Refusing update from remote peer 5aaf4595a3f74471be8a36719da5b3f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:37:41.719857 25743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5aaf4595a3f74471be8a36719da5b3f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "092f213c9e554dfab27930382e7402a5"
I20260620 11:37:41.722096 25743 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 092f213c9e554dfab27930382e7402a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5aaf4595a3f74471be8a36719da5b3f9 in term 1.
I20260620 11:37:41.719789 25769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 092f213c9e554dfab27930382e7402a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:37:41.723726 25892 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "092f213c9e554dfab27930382e7402a5" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260620 11:37:41.726619 25895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 092f213c9e554dfab27930382e7402a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aaf4595a3f74471be8a36719da5b3f9. Latest consensus state: current_term: 1 leader_uuid: "5aaf4595a3f74471be8a36719da5b3f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aaf4595a3f74471be8a36719da5b3f9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46631 } } peers { permanent_uuid: "092f213c9e554dfab27930382e7402a5" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } } peers { permanent_uuid: "2ca3db165b7e4264a1f3849cf839bca0" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } } }
I20260620 11:37:41.726670 25908 mvcc.cc:205] Tried to move back new op lower bound from 7298889571124322304 to 7298889571040452608. Current Snapshot: MvccSnapshot[applied={T|T < 7298889571053441024}]
I20260620 11:37:41.726697 25895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 092f213c9e554dfab27930382e7402a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:37:41.726737 25895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 092f213c9e554dfab27930382e7402a5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5aaf4595a3f74471be8a36719da5b3f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aaf4595a3f74471be8a36719da5b3f9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46631 } } peers { permanent_uuid: "092f213c9e554dfab27930382e7402a5" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } } peers { permanent_uuid: "2ca3db165b7e4264a1f3849cf839bca0" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } } }
I20260620 11:37:41.726776 25895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 092f213c9e554dfab27930382e7402a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:37:41.732373 25907 catalog_manager.cc:1259] Loaded cluster ID: 95696643d71c4412b56a8f368fc8c490
I20260620 11:37:41.732426 25907 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 092f213c9e554dfab27930382e7402a5: loading cluster ID for follower catalog manager: success
I20260620 11:37:41.733772 25907 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 092f213c9e554dfab27930382e7402a5: acquiring CA information for follower catalog manager: success
I20260620 11:37:41.735774 25907 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 092f213c9e554dfab27930382e7402a5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 11:37:41.737617 25774 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:41.736088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60480 (local address 127.2.243.189:41023)
0620 11:37:41.736126 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.736128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.737020 (+ 892us) server_negotiation.cc:407] Connection header received
0620 11:37:41.737117 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.737118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.737128 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.737157 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.737565 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60480: BlockingRecv error: recv got EOF from 127.0.0.1:60480 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 11:37:41.738588 25697 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:41.736303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33454 (local address 127.2.243.190:46631)
0620 11:37:41.736339 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.736341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.737816 (+ 1475us) server_negotiation.cc:407] Connection header received
0620 11:37:41.737909 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.737910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.737920 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.738077 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.738513 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33454: BlockingRecv error: recv got EOF from 127.0.0.1:33454 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 11:37:41.739253 25847 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:41.736653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47624 (local address 127.2.243.188:37245)
0620 11:37:41.736699 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.736700 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.738762 (+ 2062us) server_negotiation.cc:407] Connection header received
0620 11:37:41.738852 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.738853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.738883 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.738914 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.739195 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47624: BlockingRecv error: recv got EOF from 127.0.0.1:47624 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 11:37:41.744119 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.740698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60492 (local address 127.2.243.189:41023)
0620 11:37:41.740739 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.740741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.740895 (+ 154us) server_negotiation.cc:407] Connection header received
0620 11:37:41.740992 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.740992 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.741021 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.741062 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:41.741262 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.742065 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.743218 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.743499 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.743536 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.743758 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:41.743867 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:41.743911 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.744016 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.744051 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 11:37:41.747171 25847 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:41.746448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47636 (local address 127.2.243.188:37245)
0620 11:37:41.746527 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.746528 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.746610 (+ 82us) server_negotiation.cc:407] Connection header received
0620 11:37:41.746719 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.746720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.746736 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.746765 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.747123 (+ 358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47636: BlockingRecv error: recv got EOF from 127.0.0.1:47636 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 11:37:41.752431 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.749287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60500 (local address 127.2.243.189:41023)
0620 11:37:41.749318 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.749320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.749563 (+ 243us) server_negotiation.cc:407] Connection header received
0620 11:37:41.749651 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.749652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.749660 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.749699 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:41.749853 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.750542 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.751684 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.751972 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.752009 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.752193 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:41.752224 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:41.752251 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.752350 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.752366 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 11:37:41.755373 25847 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:41.754570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47642 (local address 127.2.243.188:37245)
0620 11:37:41.754606 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.754608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.754845 (+ 237us) server_negotiation.cc:407] Connection header received
0620 11:37:41.754998 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.754999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.755009 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.755037 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.755328 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47642: BlockingRecv error: recv got EOF from 127.0.0.1:47642 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:37:41.761487 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.758404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60506 (local address 127.2.243.189:41023)
0620 11:37:41.758437 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.758438 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.758619 (+ 181us) server_negotiation.cc:407] Connection header received
0620 11:37:41.758722 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.758723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.758732 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.758758 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:41.758947 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.759656 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.760744 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.761034 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.761069 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.761252 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:41.761282 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:41.761309 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.761407 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.761422 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 11:37:41.764261 25847 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:41.763593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47644 (local address 127.2.243.188:37245)
0620 11:37:41.763629 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.763631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.763794 (+ 163us) server_negotiation.cc:407] Connection header received
0620 11:37:41.763882 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.763883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.763893 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.763921 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.764213 (+ 292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47644: BlockingRecv error: recv got EOF from 127.0.0.1:47644 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 11:37:41.773788 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.770774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60516 (local address 127.2.243.189:41023)
0620 11:37:41.770809 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.770811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.770947 (+ 136us) server_negotiation.cc:407] Connection header received
0620 11:37:41.771043 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.771044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.771053 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.771080 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:41.771263 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.771954 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.773029 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.773317 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.773357 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.773547 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:41.773578 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:41.773606 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.773708 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.773724 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 11:37:41.781886 25847 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:41.777439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47652 (local address 127.2.243.188:37245)
0620 11:37:41.777475 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.777477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.781334 (+ 3857us) server_negotiation.cc:407] Connection header received
0620 11:37:41.781434 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.781435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.781446 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.781475 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.781838 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47652: BlockingRecv error: recv got EOF from 127.0.0.1:47652 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:37:41.794288 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.791022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60528 (local address 127.2.243.189:41023)
0620 11:37:41.791067 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.791069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.791342 (+ 273us) server_negotiation.cc:407] Connection header received
0620 11:37:41.791460 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.791461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.791470 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.791498 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:41.791664 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.792410 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.793540 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.793802 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.793840 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.794033 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:41.794064 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:41.794091 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.794204 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.794221 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 11:37:41.797389 25847 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:41.796625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47664 (local address 127.2.243.188:37245)
0620 11:37:41.796673 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.796675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.796855 (+ 180us) server_negotiation.cc:407] Connection header received
0620 11:37:41.796949 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.796950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.796960 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.796988 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.797338 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47664: BlockingRecv error: recv got EOF from 127.0.0.1:47664 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 11:37:41.817883 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.814566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60530 (local address 127.2.243.189:41023)
0620 11:37:41.814604 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.814606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.814821 (+ 215us) server_negotiation.cc:407] Connection header received
0620 11:37:41.814971 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.814972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.814983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.815010 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:41.815192 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.815923 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.817054 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.817363 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.817404 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.817607 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:41.817640 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:41.817670 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.817785 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.817803 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 11:37:41.820996 25847 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:41.820208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47676 (local address 127.2.243.188:37245)
0620 11:37:41.820244 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.820246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.820456 (+ 210us) server_negotiation.cc:407] Connection header received
0620 11:37:41.820545 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.820546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.820557 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.820585 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.820947 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47676: BlockingRecv error: recv got EOF from 127.0.0.1:47676 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:37:41.857569 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.854406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60546 (local address 127.2.243.189:41023)
0620 11:37:41.854470 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.854472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.854621 (+ 149us) server_negotiation.cc:407] Connection header received
0620 11:37:41.854678 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.854679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.854692 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.854769 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:41.854936 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.855667 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.856747 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.857031 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.857111 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.857293 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:41.857329 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:41.857389 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.857465 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.857486 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260620 11:37:41.860488 25847 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:41.859824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47688 (local address 127.2.243.188:37245)
0620 11:37:41.859877 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.859879 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.859925 (+ 46us) server_negotiation.cc:407] Connection header received
0620 11:37:41.859996 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.859997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.860009 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.860071 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.860432 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47688: BlockingRecv error: recv got EOF from 127.0.0.1:47688 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:37:41.927800 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:41.925554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.2.243.189:41023)
0620 11:37:41.925615 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.925617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.925736 (+ 119us) server_negotiation.cc:407] Connection header received
0620 11:37:41.925786 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.925786 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.925796 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.925858 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:41.925979 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.926442 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.927212 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:41.927392 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:41.927448 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:41.927605 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:41.927637 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:41.927674 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:41.927728 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 11:37:41.927742 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 11:37:41.930186 25847 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:41.929617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47700 (local address 127.2.243.188:37245)
0620 11:37:41.929665 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:41.929667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:41.929692 (+ 25us) server_negotiation.cc:407] Connection header received
0620 11:37:41.929772 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:41.929773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:41.929783 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:41.929831 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:41.930128 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47700: BlockingRecv error: recv got EOF from 127.0.0.1:47700 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:37:42.061517 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:42.059396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60566 (local address 127.2.243.189:41023)
0620 11:37:42.059452 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:42.059454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:42.059501 (+ 47us) server_negotiation.cc:407] Connection header received
0620 11:37:42.059582 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:42.059582 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:42.059591 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:42.059644 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:42.059768 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:42.060230 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:42.060962 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:42.061148 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:42.061204 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:42.061326 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:42.061354 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:42.061391 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:42.061442 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 11:37:42.061457 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260620 11:37:42.063838 25847 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:42.063164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47712 (local address 127.2.243.188:37245)
0620 11:37:42.063238 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:42.063240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:42.063309 (+ 69us) server_negotiation.cc:407] Connection header received
0620 11:37:42.063389 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:42.063390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:42.063401 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:42.063452 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:42.063768 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47712: BlockingRecv error: recv got EOF from 127.0.0.1:47712 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 11:37:42.195165 25880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5aaf4595a3f74471be8a36719da5b3f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aaf4595a3f74471be8a36719da5b3f9" member_type: VOTER last_known_addr { host: "127.2.243.190" port: 46631 } } peers { permanent_uuid: "092f213c9e554dfab27930382e7402a5" member_type: VOTER last_known_addr { host: "127.2.243.189" port: 41023 } } peers { permanent_uuid: "2ca3db165b7e4264a1f3849cf839bca0" member_type: VOTER last_known_addr { host: "127.2.243.188" port: 37245 } } }
I20260620 11:37:42.195298 25880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aaf4595a3f74471be8a36719da5b3f9 [sys.catalog]: This master's current role is: LEADER
I20260620 11:37:42.324093 25774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:42.321229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60576 (local address 127.2.243.189:41023)
0620 11:37:42.321303 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:42.321305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:42.321327 (+ 22us) server_negotiation.cc:407] Connection header received
0620 11:37:42.321358 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:42.321358 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:42.321369 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:42.321489 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:42.321716 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:42.322203 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:42.323421 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:42.323615 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:42.323673 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:42.323842 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:42.323871 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:42.323909 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:42.323992 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 11:37:42.324008 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 11:37:42.326475 25847 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:37:42.325938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47714 (local address 127.2.243.188:37245)
0620 11:37:42.326004 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:42.326007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:42.326015 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:37:42.326039 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:42.326040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:42.326057 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:42.326121 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:37:42.326430 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47714: BlockingRecv error: recv got EOF from 127.0.0.1:47714 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 11:37:42.648445 25861 catalog_manager.cc:1259] Loaded cluster ID: 95696643d71c4412b56a8f368fc8c490
I20260620 11:37:42.648510 25861 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2ca3db165b7e4264a1f3849cf839bca0: loading cluster ID for follower catalog manager: success
I20260620 11:37:42.649293 25861 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2ca3db165b7e4264a1f3849cf839bca0: acquiring CA information for follower catalog manager: success
I20260620 11:37:42.649591 25861 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2ca3db165b7e4264a1f3849cf839bca0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 11:37:42.842330 26028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:42.839782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40106 (local address 127.2.243.189:41023)
0620 11:37:42.839899 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:42.839902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:42.839980 (+ 78us) server_negotiation.cc:407] Connection header received
0620 11:37:42.840095 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:42.840095 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:42.840106 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:42.840177 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:42.840317 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:42.840812 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:42.841584 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:42.841799 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:42.841881 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:42.842047 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:42.842085 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:42.842145 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:42.842206 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 11:37:42.842230 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
I20260620 11:37:42.846177 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:42.843879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46926 (local address 127.2.243.188:37245)
0620 11:37:42.843979 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:42.843980 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:42.843988 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:37:42.844014 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:42.844015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:42.844082 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:42.844151 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:42.844236 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:42.844700 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:42.845460 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:42.845661 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:42.845714 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:42.845847 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:42.845952 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:42.846010 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:42.846067 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 11:37:42.846095 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":54,"threads_started":1}
I20260620 11:37:42.850682 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:42.848076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46932 (local address 127.2.243.188:37245)
0620 11:37:42.848122 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:42.848125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:42.848215 (+ 90us) server_negotiation.cc:407] Connection header received
0620 11:37:42.848286 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:42.848287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:42.848297 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:42.848345 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:42.848511 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:42.849225 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:42.849939 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:42.850217 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:42.850280 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:42.850412 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:42.850445 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:42.850492 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:42.850590 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 11:37:42.850619 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 11:37:42.850798 26028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:42.847866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40120 (local address 127.2.243.189:41023)
0620 11:37:42.847903 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:42.847906 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:42.848139 (+ 233us) server_negotiation.cc:407] Connection header received
0620 11:37:42.848195 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:42.848196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:42.848206 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:42.848256 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:42.848394 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:42.848951 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:42.849977 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:42.850296 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:42.850353 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:42.850508 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:42.850551 (+ 43us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:42.850581 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:42.850694 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 11:37:42.850717 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:37:42.851050 26041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:37:42.847828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.2.243.190:46631)
0620 11:37:42.847943 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 11:37:42.847944 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:37:42.847951 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:37:42.848020 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:37:42.848021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:37:42.848075 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:37:42.848200 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:37:42.848356 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:42.849155 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:42.850115 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:37:42.850400 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:37:42.850471 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:37:42.850671 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:37:42.850787 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:37:42.850855 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 11:37:42.850941 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 11:37:42.850964 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":57,"threads_started":1}
I20260620 11:37:42.852926 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 25626
I20260620 11:37:42.858525 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 25701
I20260620 11:37:42.863327 3022 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDepOyZ/build/release/bin/kudu with pid 25776
2026-06-20T11:37:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2812 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5585 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (270042 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST